[jira] [Reopened] (HBASE-17056) Remove checked in PB generated files

2017-07-06 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-17056:
---

Reverting for now. I am away from computer for a while so unable to provide the 
support to get over any issues this large commit generates. Will try again 
later when build settles. It is failing currently w/ OOMEs.

> Remove checked in PB generated files 
> -
>
> Key: HBASE-17056
> URL: https://issues.apache.org/jira/browse/HBASE-17056
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Enis Soztutar
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 
> 0002-HBASE-17056-Remove-checked-in-PB-generated-files.patch, 
> HBASE-17056.master.001.patch, HBASE-17056.master.002.patch, 
> HBASE-17056.master.003.patch
>
>
> Now that we have the new PB maven plugin, there is no need to have the PB 
> files checked in to the repo. The reason we did that was to ease up developer 
> env setup. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18330) NPE in ReplicationZKLockCleanerChore

2017-07-06 Thread Minwoo Kang (JIRA) 
Minwoo Kang created HBASE-18330:
---

 Summary: NPE in ReplicationZKLockCleanerChore
 Key: HBASE-18330
 URL: https://issues.apache.org/jira/browse/HBASE-18330
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.5
Reporter: Minwoo Kang
Priority: Minor


While I am watching HMaster log, I found NullPointerException Logs.



2017-07-06 09:05:02,579 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive
2017-07-06 09:05:02,585 ERROR [,1498445640728_ChoreService_2] 
hbase.ScheduledChore: Caught error
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.master.cleaner.ReplicationZKLockCleanerChore.chore(ReplicationZKLockCleanerChore.java:80)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-07-06 09:05:02,585 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive
2017-07-06 09:05:02,586 DEBUG [,1498445640728_ChoreService_1] 
cleaner.CleanerChore: Removing: hdfs://*** from archive



Here is related code:
  List replicators = queues.getListOfReplicators();
  for (String replicator: replicators) {



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HBASE-16574) Add backup / restore feature to refguide

2017-07-06 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HBASE-16574:


Looks like this missed the merge to master branch.

> Add backup / restore feature to refguide
> 
>
> Key: HBASE-16574
> URL: https://issues.apache.org/jira/browse/HBASE-16574
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Frank Welsch
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, 
> HBASE-16574.001.patch, HBASE-16574.002.patch, hbase_reference_guide.v1.pdf
>
>
> This issue is to add backup / restore feature description to hbase refguide.
> The description should cover:
> scenarios where backup / restore is used
> backup / restore commands and sample usage
> considerations in setup



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HBASE-16458) Shorten backup / restore test execution time

2017-07-06 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov reopened HBASE-16458:
---

> Shorten backup / restore test execution time
> 
>
> Key: HBASE-16458
> URL: https://issues.apache.org/jira/browse/HBASE-16458
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: 16458.HBASE-7912.v3.txt, 16458.HBASE-7912.v4.txt, 
> 16458.HBASE-7912.v5.txt, 16458.v1.txt, 16458.v2.txt, 16458.v3.txt
>
>
> Below was timing information for all the backup / restore tests (today's 
> result):
> {code}
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 576.273 sec - 
> in org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.67 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.34 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Running org.apache.hadoop.hbase.backup.TestBackupAdmin
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.251 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupAdmin
> Running org.apache.hadoop.hbase.backup.TestHFileArchiving
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.323 sec - 
> in org.apache.hadoop.hbase.backup.TestHFileArchiving
> Running org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.492 sec - 
> in org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Running org.apache.hadoop.hbase.backup.TestBackupDescribe
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.758 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupDescribe
> Running org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.187 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 330.539 sec - 
> in org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Running org.apache.hadoop.hbase.backup.TestRemoteBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.371 sec - 
> in org.apache.hadoop.hbase.backup.TestRemoteBackup
> Running org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.893 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Running org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.779 sec - 
> in org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.815 sec - 
> in org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Running org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.517 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Running org.apache.hadoop.hbase.backup.TestRemoteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.799 sec - 
> in org.apache.hadoop.hbase.backup.TestRemoteRestore
> Running org.apache.hadoop.hbase.backup.TestFullRestore
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 317.711 sec 
> - in org.apache.hadoop.hbase.backup.TestFullRestore
> Running org.apache.hadoop.hbase.backup.TestFullBackupSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.045 sec - 
> in org.apache.hadoop.hbase.backup.TestFullBackupSet
> Running org.apache.hadoop.hbase.backup.TestBackupDelete
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.214 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupDelete
> Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.631 sec - 
> in org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 190.358 sec - 
> in org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
> Running 
> org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination
> Tests run: 3, Failures: 0, Errors: 0,

Still Failing: HBase Generate Website

2017-07-06 Thread Apache Jenkins Server 
Build status: Still Failing

The HBase website has not been updated to incorporate HBase commit 
${HBASE_GIT_SHA}.

See https://builds.apache.org/job/hbase_generate_website/1046/console

[jira] [Created] (HBASE-18329) update links in config guide to point to java 8 references

2017-07-06 Thread Artem Ervits (JIRA) 
Artem Ervits created HBASE-18329:


 Summary: update links in config guide to point to java 8 references
 Key: HBASE-18329
 URL: https://issues.apache.org/jira/browse/HBASE-18329
 Project: HBase
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.0.0-alpha-1, 1.1.11, 1.2.6, 1.3.1
Reporter: Artem Ervits
Assignee: Artem Ervits
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

2017-07-06 Thread Sean Busbey 
that sounds like our project structure is broken. Please make sure there's
a jira that tracks it and I'll take a look later.

On Thu, Jul 6, 2017 at 6:15 PM, Stack  wrote:

> I tried publishing hbase-3.0.0-SNAPSHOT... so hbase-checkstyle was up in
> repo (presuming it relied on an aged-out snapshot). Seems to have 'fixed'
> it for now
>
> St.Ack
>
> On Thu, Jul 6, 2017 at 12:50 PM, Stack  wrote:
>
> > The 3.0.0-SNAPSHOT looks suspicious ... the hbase version
> > St.Ack
> >
> > On Thu, Jul 6, 2017 at 12:49 PM, Stack  wrote:
> >
> >> On Thu, Jul 6, 2017 at 12:48 PM, Stack  wrote:
> >>
> >>> Checkstyle is currently broke on our builds... looking.
> >>> St.Ack
> >>>
> >>>
> >> Works if I run it locally (of course)
> >> St.Ack
> >>
> >>
> >>
> >>
> >>>
> >>>
> >>> [ERROR] Failed to execute goal org.apache.maven.plugins:
> maven-checkstyle-plugin:2.17:checkstyle (default-cli) on project hbase:
> Execution default-cli of goal org.apache.maven.plugins:
> maven-checkstyle-plugin:2.17:checkstyle failed: Plugin
> org.apache.maven.plugins:maven-checkstyle-plugin:2.17 or one of its
> dependencies could not be resolved: Could not find artifact
> org.apache.hbase:hbase-checkstyle:jar:3.0.0-SNAPSHOT in Nexus (
> http://repository.apache.org/snapshots) -> [Help 1][ERROR] [ERROR] To see
> the full stack trace of the errors, re-run Maven with the -e switch.[ERROR]
> Re-run Maven using the -X switch to enable full debug logging.[ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:[ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/
> PluginResolutionExceptionBuild step 'Invoke top-level Maven targets'
> marked build as failure
> >>> Performing Post build task...
> >>> Match found for :.* : True
> >>> Logical operation result is TRUE
> >>> Running script  : # Run zombie detector script
> >>> ./dev-support/zombie-detector.sh --jenkins ${BUILD_ID}
> >>> [a3159d73] $ /bin/bash -xe /tmp/hudson1697041977582083402.sh
> >>> + ./dev-support/zombie-detector.sh --jenkins 3320
> >>> Thu Jul  6 01:37:09 UTC 2017 We're ok: there is no zombie test
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jun 30, 2017 at 2:43 PM, Sean Busbey 
> wrote:
> >>>
>  jacoco was added ages ago. I'd guess that something changed on the
>  machines
>  we use to cause it to stop working.
> 
>  On Thu, Jun 29, 2017 at 12:02 PM, Stack  wrote:
> 
>  > On Wed, Jun 28, 2017 at 8:43 AM, Josh Elser 
>  wrote:
>  >
>  > >
>  > >
>  > > On 6/27/17 7:20 PM, Stack wrote:
>  > >
>  > >> * test-patch's whitespace plugin can configured to ignore some
>  files
>  > (but
>  > >>> I
>  > >>> can't think of any we'd care to so whitelist)
>  > >>>
>  > >>> Generated files.
>  > >>
>  > >
>  > > Oh my goodness, yes, please. This has been such a pain in the rear
>  for me
>  > > as I've been rebasing space quota patches. Sometimes, the spaces
> in
>  > > pb-gen'ed code are removed by folks before commit, other times
> they
>  > aren't.
>  > >
>  >
>  > Agree sir. Its a distraction at least.
>  >
>  > I see Jacoco report here now:
>  > https://builds.apache.org/job/HBase-Trunk_matrix/jdk=JDK%
>  > 201.8%20(latest),label=Hadoop/3277/
>  >
>  > Maybe it has been there always and I just haven't noticed.
>  >
>  > Its all 0%. We need to turn on stuff?
>  >
>  > St.Ack
>  >
> 
> >>>
> >>>
> >>
> >
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

2017-07-06 Thread Stack 
I tried publishing hbase-3.0.0-SNAPSHOT... so hbase-checkstyle was up in
repo (presuming it relied on an aged-out snapshot). Seems to have 'fixed'
it for now

St.Ack

On Thu, Jul 6, 2017 at 12:50 PM, Stack  wrote:

> The 3.0.0-SNAPSHOT looks suspicious ... the hbase version
> St.Ack
>
> On Thu, Jul 6, 2017 at 12:49 PM, Stack  wrote:
>
>> On Thu, Jul 6, 2017 at 12:48 PM, Stack  wrote:
>>
>>> Checkstyle is currently broke on our builds... looking.
>>> St.Ack
>>>
>>>
>> Works if I run it locally (of course)
>> St.Ack
>>
>>
>>
>>
>>>
>>>
>>> [ERROR] Failed to execute goal 
>>> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle 
>>> (default-cli) on project hbase: Execution default-cli of goal 
>>> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle failed: 
>>> Plugin org.apache.maven.plugins:maven-checkstyle-plugin:2.17 or one of its 
>>> dependencies could not be resolved: Could not find artifact 
>>> org.apache.hbase:hbase-checkstyle:jar:3.0.0-SNAPSHOT in Nexus 
>>> (http://repository.apache.org/snapshots) -> [Help 1][ERROR] [ERROR] To see 
>>> the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] 
>>> Re-run Maven using the -X switch to enable full debug logging.[ERROR] 
>>> [ERROR] For more information about the errors and possible solutions, 
>>> please read the following articles:[ERROR] [Help 1] 
>>> http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionExceptionBuild
>>>  step 'Invoke top-level Maven targets' marked build as failure
>>> Performing Post build task...
>>> Match found for :.* : True
>>> Logical operation result is TRUE
>>> Running script  : # Run zombie detector script
>>> ./dev-support/zombie-detector.sh --jenkins ${BUILD_ID}
>>> [a3159d73] $ /bin/bash -xe /tmp/hudson1697041977582083402.sh
>>> + ./dev-support/zombie-detector.sh --jenkins 3320
>>> Thu Jul  6 01:37:09 UTC 2017 We're ok: there is no zombie test
>>>
>>>
>>>
>>>
>>> On Fri, Jun 30, 2017 at 2:43 PM, Sean Busbey  wrote:
>>>
 jacoco was added ages ago. I'd guess that something changed on the
 machines
 we use to cause it to stop working.

 On Thu, Jun 29, 2017 at 12:02 PM, Stack  wrote:

 > On Wed, Jun 28, 2017 at 8:43 AM, Josh Elser 
 wrote:
 >
 > >
 > >
 > > On 6/27/17 7:20 PM, Stack wrote:
 > >
 > >> * test-patch's whitespace plugin can configured to ignore some
 files
 > (but
 > >>> I
 > >>> can't think of any we'd care to so whitelist)
 > >>>
 > >>> Generated files.
 > >>
 > >
 > > Oh my goodness, yes, please. This has been such a pain in the rear
 for me
 > > as I've been rebasing space quota patches. Sometimes, the spaces in
 > > pb-gen'ed code are removed by folks before commit, other times they
 > aren't.
 > >
 >
 > Agree sir. Its a distraction at least.
 >
 > I see Jacoco report here now:
 > https://builds.apache.org/job/HBase-Trunk_matrix/jdk=JDK%
 > 201.8%20(latest),label=Hadoop/3277/
 >
 > Maybe it has been there always and I just haven't noticed.
 >
 > Its all 0%. We need to turn on stuff?
 >
 > St.Ack
 >

>>>
>>>
>>
>

Re: Plan for Hybrid Logical Clocks

2017-07-06 Thread Amit Patel 
Hi everyone,

As an update, the core HLC work has been pushed out as an initial commit to
a public branch called HBASE-14070.HLC
.
Details regarding the core HLC commit can be viewed at HBASE-18305
. The next step will be
another commit on that branch that would undo using the master's timestamp
for meta updates. The progress for this task will be tracked in HBASE-18328
.

On Fri, Jun 23, 2017 at 10:38 AM, Enis Söztutar  wrote:

> The plan looks good.
>
> >However, if it is done before HBase 2.0.0 beta and if it is proven to be
> solid, then we can discuss bringing it back to 2.0.0.
> It will be pretty good to have at least the base in 2.0, so that we can
> later enable HLC for user tables in later releases. Of course, it depends
> sufficient testing.
>
> Enis
>
> On Fri, Jun 23, 2017 at 10:03 AM, Amit Patel 
> wrote:
>
> > For now, yes, that is the plan. However, if it is done before HBase 2.0.0
> > beta and if it is proven to be solid, then we can discuss bringing it
> back
> > to 2.0.0.
> >
> > On Fri, Jun 23, 2017 at 6:12 AM, Sean Busbey  wrote:
> >
> > > It'd be great to see this land. You only mention the master branch as
> > > a merge target; is the assumption that this would only be a HBase 3.0+
> > > addition?
> > >
> > > On Thu, Jun 22, 2017 at 7:34 PM, Amit Patel 
> > > wrote:
> > > > Hi everyone,
> > > >
> > > >
> > > >
> > > > I'm Amit and I've been picking up on the past work on Hybrid Logical
> > > Clocks
> > > > (HBASE-14070 )
> that
> > > was
> > > > done by Sai Teja Ranuva last summer. The most recent status of HLC on
> > > HBase
> > > > can be found here
> > > >  > > NuF8TbQkZiZCwFhbHZ7JXRjg3oA/edit#>
> > > > and a prior document by Enis Soztutar with discussion can be found
> here
> > > >  > > bXy8P9h6kWC05Bhw/edit#>.
> > > > I think the effort is at a point where it makes sense to create a
> > public
> > > > branch.
> > > >
> > > >
> > > >
> > > > Currently the plan is to create a public branch and incrementally add
> > on
> > > > functionality and tests. Initially, core HLC would be introduced as a
> > > > commit. From there, HLC would then be enabled for just the meta table
> > > > (updating the clock on events like region open/close, recovery,
> > > > replication, etc). Once we are fully confident that HLC at least
> works
> > on
> > > > the meta table then I anticipate it would be appropriate to merge
> with
> > > the
> > > > master branch.
> > > >
> > > >
> > > >
> > > > Follow up commits would extend the effort by:
> > > >
> > > >
> > > >
> > > >-
> > > >
> > > >Addition of a protobuf message called NodeTime that holds
> timestamp
> > > >corresponding to a send event between nodes.
> > > >-
> > > >
> > > >Addition of NodeTime as a field to messages sent between nodes for
> > > >messages like requests and responses for region open/close.
> > > >-
> > > >
> > > >Addition of integration tests that verify HLC works correctly with
> > > >recovery, replication, region open, and region close.
> > > >-
> > > >
> > > >Later enabling HLC on user tables (main barrier are tests that
> rely
> > on
> > > >client-side setting of timestamps for table mutations which would
> no
> > > longer
> > > >be allowed on an HLC table).
> > > >
> > > >
> > > >
> > > > Work that I have implemented under the outstanding work
> > > >  > > NuF8TbQkZiZCwFhbHZ7JXRjg3oA/edit#heading=h.8x5d8iakmo8b>
> > > > as a part of getting HLC working for user tables includes:
> > > >
> > > >
> > > >
> > > >-
> > > >
> > > >Mapping of time ranges from client GETs
> > > >-
> > > >
> > > >Disallowing clients to set timestamps for HLC and System Monotonic
> > > tables
> > > >
> > > >
> > > >
> > > > Feel free to chime in with any comments, suggestions, or other input.
> > > >
> > > > --
> > > > Amit
> > >
> >
> >
> >
> > --
> > Amit
> >
>



-- 
Amit

[jira] [Created] (HBASE-18328) Undoing using the master's timestamp for meta updates

2017-07-06 Thread Amit Patel (JIRA) 
Amit Patel created HBASE-18328:
--

 Summary: Undoing using the master's timestamp for meta updates
 Key: HBASE-18328
 URL: https://issues.apache.org/jira/browse/HBASE-18328
 Project: HBase
  Issue Type: Sub-task
Reporter: Amit Patel
Assignee: Amit Patel
Priority: Minor


After introducing [the Core HLC 
framework|https://issues.apache.org/jira/browse/HBASE-18305], the next step is 
to remove all instances where the timestamp is explicitly set for meta updates. 
Most of these changes take place in MetaTableAccessor. The patch for this task 
will be going against the HBASE-14070.HLC branch.

Credit for this work all goes to our [~saitejar].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18305) Initial patch (Core HLC) for public branch

2017-07-06 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-18305.
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HBASE-14070
 Release Note: 
Adds Core HLC framework (i.e., Clock, Timestamp API) and tests. Practically all 
of this work has been done by our [~saitejar]. 

This patch in particular intentionally does not include:
 * undoing using the master's timestamp for meta updates,
 * enabling HLC for meta
 * updating the clock for events like recovery, replication, region open/close. 


Pushed to HBASE-10470.HLC branch. Thanks for the patch [~amit.patel]!

> Initial patch (Core HLC) for public branch
> --
>
> Key: HBASE-18305
> URL: https://issues.apache.org/jira/browse/HBASE-18305
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amit Patel
>Assignee: Amit Patel
> Fix For: HBASE-14070
>
> Attachments: 0001-HBASE-14070-Core-HLC-revision-1.patch, 
> HBASE-18305.HBASE-14070.HLC.001.patch, HBASE-18305.master.001.patch, 
> HBASE-18305.master.002.patch, HBASE-18305.master.003.patch
>
>
> Used for tracking the status of the initial patch for HLC. Core HLC refers to 
> the addition of the HLC framework (i.e., Clock, Timestamp API), integration 
> of said framework with the codebase, and changes to tests. Practically all of 
> this work has been done by our [~saitejar]. 
> This patch in particular intentionally does not include:
> * undoing using the master's timestamp for meta updates,
> * enabling HLC for meta
> * updating the clock for events like recovery, replication, region 
> open/close. 
> These changes will be introduced soon in smaller followup commits.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

2017-07-06 Thread Stack 
On Thu, Jul 6, 2017 at 12:48 PM, Stack  wrote:

> Checkstyle is currently broke on our builds... looking.
> St.Ack
>
>
Works if I run it locally (of course)
St.Ack




>
>
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle 
> (default-cli) on project hbase: Execution default-cli of goal 
> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle failed: 
> Plugin org.apache.maven.plugins:maven-checkstyle-plugin:2.17 or one of its 
> dependencies could not be resolved: Could not find artifact 
> org.apache.hbase:hbase-checkstyle:jar:3.0.0-SNAPSHOT in Nexus 
> (http://repository.apache.org/snapshots) -> [Help 1][ERROR] [ERROR] To see 
> the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] 
> Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] 
> For more information about the errors and possible solutions, please read the 
> following articles:[ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionExceptionBuild
>  step 'Invoke top-level Maven targets' marked build as failure
> Performing Post build task...
> Match found for :.* : True
> Logical operation result is TRUE
> Running script  : # Run zombie detector script
> ./dev-support/zombie-detector.sh --jenkins ${BUILD_ID}
> [a3159d73] $ /bin/bash -xe /tmp/hudson1697041977582083402.sh
> + ./dev-support/zombie-detector.sh --jenkins 3320
> Thu Jul  6 01:37:09 UTC 2017 We're ok: there is no zombie test
>
>
>
>
> On Fri, Jun 30, 2017 at 2:43 PM, Sean Busbey  wrote:
>
>> jacoco was added ages ago. I'd guess that something changed on the
>> machines
>> we use to cause it to stop working.
>>
>> On Thu, Jun 29, 2017 at 12:02 PM, Stack  wrote:
>>
>> > On Wed, Jun 28, 2017 at 8:43 AM, Josh Elser  wrote:
>> >
>> > >
>> > >
>> > > On 6/27/17 7:20 PM, Stack wrote:
>> > >
>> > >> * test-patch's whitespace plugin can configured to ignore some files
>> > (but
>> > >>> I
>> > >>> can't think of any we'd care to so whitelist)
>> > >>>
>> > >>> Generated files.
>> > >>
>> > >
>> > > Oh my goodness, yes, please. This has been such a pain in the rear
>> for me
>> > > as I've been rebasing space quota patches. Sometimes, the spaces in
>> > > pb-gen'ed code are removed by folks before commit, other times they
>> > aren't.
>> > >
>> >
>> > Agree sir. Its a distraction at least.
>> >
>> > I see Jacoco report here now:
>> > https://builds.apache.org/job/HBase-Trunk_matrix/jdk=JDK%
>> > 201.8%20(latest),label=Hadoop/3277/
>> >
>> > Maybe it has been there always and I just haven't noticed.
>> >
>> > Its all 0%. We need to turn on stuff?
>> >
>> > St.Ack
>> >
>>
>
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

2017-07-06 Thread Stack 
Checkstyle is currently broke on our builds... looking.
St.Ack



[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle
(default-cli) on project hbase: Execution default-cli of goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle
failed: Plugin org.apache.maven.plugins:maven-checkstyle-plugin:2.17
or one of its dependencies could not be resolved: Could not find
artifact org.apache.hbase:hbase-checkstyle:jar:3.0.0-SNAPSHOT in Nexus
(http://repository.apache.org/snapshots) -> [Help 1][ERROR] [ERROR] To
see the full stack trace of the errors, re-run Maven with the -e
switch.[ERROR] Re-run Maven using the -X switch to enable full debug
logging.[ERROR] [ERROR] For more information about the errors and
possible solutions, please read the following articles:[ERROR] [Help
1] 
http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionExceptionBuild
step 'Invoke top-level Maven targets' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script  : # Run zombie detector script
./dev-support/zombie-detector.sh --jenkins ${BUILD_ID}
[a3159d73] $ /bin/bash -xe /tmp/hudson1697041977582083402.sh
+ ./dev-support/zombie-detector.sh --jenkins 3320
Thu Jul  6 01:37:09 UTC 2017 We're ok: there is no zombie test




On Fri, Jun 30, 2017 at 2:43 PM, Sean Busbey  wrote:

> jacoco was added ages ago. I'd guess that something changed on the machines
> we use to cause it to stop working.
>
> On Thu, Jun 29, 2017 at 12:02 PM, Stack  wrote:
>
> > On Wed, Jun 28, 2017 at 8:43 AM, Josh Elser  wrote:
> >
> > >
> > >
> > > On 6/27/17 7:20 PM, Stack wrote:
> > >
> > >> * test-patch's whitespace plugin can configured to ignore some files
> > (but
> > >>> I
> > >>> can't think of any we'd care to so whitelist)
> > >>>
> > >>> Generated files.
> > >>
> > >
> > > Oh my goodness, yes, please. This has been such a pain in the rear for
> me
> > > as I've been rebasing space quota patches. Sometimes, the spaces in
> > > pb-gen'ed code are removed by folks before commit, other times they
> > aren't.
> > >
> >
> > Agree sir. Its a distraction at least.
> >
> > I see Jacoco report here now:
> > https://builds.apache.org/job/HBase-Trunk_matrix/jdk=JDK%
> > 201.8%20(latest),label=Hadoop/3277/
> >
> > Maybe it has been there always and I just haven't noticed.
> >
> > Its all 0%. We need to turn on stuff?
> >
> > St.Ack
> >
>

[jira] [Created] (HBASE-18327) redo test-patch personality 'hadoopcheck' to better account for feature branches

2017-07-06 Thread Sean Busbey (JIRA) 
Sean Busbey created HBASE-18327:
---

 Summary: redo test-patch personality 'hadoopcheck' to better 
account for feature branches
 Key: HBASE-18327
 URL: https://issues.apache.org/jira/browse/HBASE-18327
 Project: HBase
  Issue Type: Improvement
  Components: build, test
Reporter: Sean Busbey
Priority: Minor


right now our 'which hadoop checks do we need' check looks like this:

{code}
  if [[ "${PATCH_BRANCH}" = "master" ]]; then
hbase_hadoop2_versions=${HBASE_MASTER_HADOOP2_VERSIONS}
hbase_hadoop3_versions=${HBASE_MASTER_HADOOP3_VERSIONS}
  elif [[ ${PATCH_BRANCH} = branch-2* ]]; then
hbase_hadoop2_versions=${HBASE_BRANCH2_HADOOP2_VERSIONS}
hbase_hadoop3_versions=${HBASE_BRANCH2_HADOOP3_VERSIONS}
  else
hbase_hadoop2_versions=${HBASE_HADOOP2_VERSIONS}
hbase_hadoop3_versions=${HBASE_HADOOP3_VERSIONS}
  fi
{code}

the check is basically "if master do this, if like branch-2 do that, otherwise 
behave like branch-1".

we often have feature branches that thus end up being treated like branch-1, 
even though those branches should all be based off of master. (since we follow 
a master-first development approach.)

we should redo this check so it's "if branch-1 do this, if branch-2 do that, 
otherwise behave like master"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-06 Thread Thiruvel Thirumoolan 
Congratulations Devaraj!
-Thiruvel


On Wednesday, July 5, 2017, 2:53:02 PM PDT, Jerry He  wrote:

Congrats, Devaraj !


Thanks,

Jerry

On Wed, Jul 5, 2017 at 1:36 PM, Nick Dimiduk  wrote:
> Congratulations Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
>> I'm pleased to announce yet another PMC addition in the form of Devaraj
>> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
>> long-standing member in our community. We all look forward to the continued
>> contributions and project leadership.
>>
>> Please join me in welcoming Devaraj!
>>
>> - Josh (on behalf of the PMC)
>>

[jira] [Resolved] (HBASE-12211) hbase-daemon.sh direct the ulimit output to log .out instead of log .log?

2017-07-06 Thread Mike Drob (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-12211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob resolved HBASE-12211.
---
Resolution: Not A Problem

There was no discussion on the mailing list about this topic, unlikely to get 
picked up anymore.

> hbase-daemon.sh direct the ulimit output to log .out instead of log .log?
> -
>
> Key: HBASE-12211
> URL: https://issues.apache.org/jira/browse/HBASE-12211
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: stanley shi
>Priority: Minor
>
> In the hbase-daemon.sh file line 199, it has the content:
> {quote}echo "`ulimit -a`" >> $loglog 2>&1{quote}
> The variable loglog is defined as:
> {quote}
> logout=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.out
> loggc=$HBASE_LOG_DIR/$HBASE_LOG_PREFIX.gc
> loglog="${HBASE_LOG_DIR}/${HBASE_LOGFILE}"
> {quote}
> For my understanding, this information should be printed to the "logout" 
> variable; we should not mix this "ulimit" information with the actual log 
> printed by hbase java program.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] More Shading

2017-07-06 Thread Stack 
FYI:

hbase-thirdparty has had its first release. Yesterday I
committed HBASE-17056 "Remove checked in PB generated files" which apart
from purging all checked-in generated files (30MB), it moves our hbase core
(master and branch-2) to start using the thirdparty jar.

Things might be interesting over next few days so shout if you run into
issues.

One issue is the need for mvn install where before mvn compile might have
been enough (see below for example of the issue you'd see now if you did
mvn clean compile only). We need the mvn install because we need to shade
the generated files so they use the relocated protobuf; shade happens after
we've made a module jar.

IDEs will complain too if they pick up generated src from target dirs since
they'll be unsatisfied references to protobuf3 objects -- unless you point
at the shaded jar. Let me see if I can do something about the latter.

Let me know if HBASE-17056 is too much for devs to bear. Can always revert
(though nice having the protobuf generation in-line w/ the build). It is
sort of a side-benefit on the general shading project so could back it out
and still have the shading of netty, guava, protobuf, etc.

Thanks,

St.Ack

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
(default-compile) on project hbase-procedure: Compilation failure:
Compilation failure:
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11]
cannot access com.google.protobuf.GeneratedMessageV3
[ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7]
cannot access com.google.protobuf.GeneratedMessageV3.Builder
[ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder not
found
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22]
cannot find symbol
[ERROR] symbol:   method
writeDelimitedTo(org.apache.hadoop.fs.FSDataOutputStream)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureStoreTracker
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20]
cannot find symbol
[ERROR] symbol:   method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20]
cannot find symbol
[ERROR] symbol:   method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20]
cannot find symbol
[ERROR] symbol:   method
writeDelimitedTo(org.apache.hadoop.hbase.procedure2.util.ByteSlot)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureWALEntry
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30]
cannot find symbol
[ERROR] symbol:   method toByteArray()
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ErrorHandlingProtos.ForeignExceptionMessage
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17]
cannot find symbol
[ERROR] symbol:   method writeDelimitedTo(java.io.OutputStream)
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.StateMachineProcedureData
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString cannot be
converted to com.google.protobuf.ByteString
[ERROR]
/home/stack/hbase.git/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17]
cannot find symbol
[ERROR] symbol:   method writeDelimitedTo(java.io.OutputStream)
[ERROR] location: class

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

2017-07-06 Thread Enis Söztutar 
Congrats!

Enis

On Thu, Jul 6, 2017 at 10:57 AM, Devaraj Das  wrote:

> Congratulations, Chunhui!
> 
> From: Yu Li 
> Sent: Monday, July 03, 2017 10:24 PM
> To: dev@hbase.apache.org; Hbase-User
> Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
>
> On behalf of the Apache HBase PMC I am pleased to announce that Chunhui
> Shen
> has accepted our invitation to become a PMC member on the Apache
> HBase project. He has been an active contributor to HBase for past many
> years. Looking forward for many more contributions from him.
>
> Please join me in welcoming Chunhui to the HBase PMC!
>
> Best Regards,
> Yu
>
>
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-06 Thread Enis Söztutar 
Congrats!

Enis

On Thu, Jul 6, 2017 at 10:55 AM, Devaraj Das  wrote:

> Thanks, everyone!!
> 
> From: Phil Yang 
> Sent: Thursday, July 06, 2017 1:45 AM
> To: HBase Dev List
> Subject: Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC
>
> Congratulations!
>
> Thanks,
> Phil
>
>
> 2017-07-06 15:39 GMT+08:00 Chia-Ping Tsai :
>
> > Congratulations!!!
> >
> > On 2017-07-06 00:27 (+0800), Josh Elser  wrote:
> > > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > > long-standing member in our community. We all look forward to the
> > > continued contributions and project leadership.
> > >
> > > Please join me in welcoming Devaraj!
> > >
> > > - Josh (on behalf of the PMC)
> > >
> >
>
>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

2017-07-06 Thread Devaraj Das 
Congratulations, Chunhui!

From: Yu Li 
Sent: Monday, July 03, 2017 10:24 PM
To: dev@hbase.apache.org; Hbase-User
Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

On behalf of the Apache HBase PMC I am pleased to announce that Chunhui Shen
has accepted our invitation to become a PMC member on the Apache
HBase project. He has been an active contributor to HBase for past many
years. Looking forward for many more contributions from him.

Please join me in welcoming Chunhui to the HBase PMC!

Best Regards,
Yu

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-06 Thread Devaraj Das 
Thanks, everyone!! 

From: Phil Yang 
Sent: Thursday, July 06, 2017 1:45 AM
To: HBase Dev List
Subject: Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

Congratulations!

Thanks,
Phil


2017-07-06 15:39 GMT+08:00 Chia-Ping Tsai :

> Congratulations!!!
>
> On 2017-07-06 00:27 (+0800), Josh Elser  wrote:
> > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > long-standing member in our community. We all look forward to the
> > continued contributions and project leadership.
> >
> > Please join me in welcoming Devaraj!
> >
> > - Josh (on behalf of the PMC)
> >
>

Failure: HBase Generate Website

2017-07-06 Thread Apache Jenkins Server 
Build status: Failure

The HBase website has not been updated to incorporate HBase commit 
${HBASE_GIT_SHA}.

See https://builds.apache.org/job/hbase_generate_website/1045/console

[jira] [Resolved] (HBASE-12542) Delete a family of table online will crash regionserver

2017-07-06 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-12542.
---
  Resolution: Cannot Reproduce
Hadoop Flags: Reviewed

Thanks [~psomogyi] for investigating. Closing... as 'Cannot Reproduce'.

> Delete a family of table online will crash regionserver 
> 
>
> Key: HBASE-12542
> URL: https://issues.apache.org/jira/browse/HBASE-12542
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Liu Shaohui
>Assignee: Peter Somogyi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-12542-v1.diff
>
>
> Using alter command to delete a family of table online will make the 
> regionsevers that serve the regions of the table crash.
> {code}
> alter 't', NAME => 'f', METHOD => 'delete'
> {code}
> The reason is that TableDeleteFamilyHandler in HMaster delete the family dir 
> firstly and then reopen all the regions of table.
> When the regionserver reopen the region, it will crash for the exception in 
> flushing memstore to hfile of the deleted family during closing the region, 
> because the parent dir of the hfile has been deleted in 
> TableDeleteFamilyHandler.
> See: TableDeleteFamilyHandler.java #57
> A simple solution is change the order of operations in 
> TableDeleteFamilyHandler.
> - update table descriptor first, 
> - reopen all the regions,
> - delete the the family dir at last.
> Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-06 Thread Phil Yang 
Congratulations!

Thanks,
Phil


2017-07-06 15:39 GMT+08:00 Chia-Ping Tsai :

> Congratulations!!!
>
> On 2017-07-06 00:27 (+0800), Josh Elser  wrote:
> > I'm pleased to announce yet another PMC addition in the form of Devaraj
> > Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
> > long-standing member in our community. We all look forward to the
> > continued contributions and project leadership.
> >
> > Please join me in welcoming Devaraj!
> >
> > - Josh (on behalf of the PMC)
> >
>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

2017-07-06 Thread Chia-Ping Tsai 
Congratulations!!!

On 2017-07-06 00:27 (+0800), Josh Elser  wrote: 
> I'm pleased to announce yet another PMC addition in the form of Devaraj 
> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a 
> long-standing member in our community. We all look forward to the 
> continued contributions and project leadership.
> 
> Please join me in welcoming Devaraj!
> 
> - Josh (on behalf of the PMC)
>

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-06 Thread 宾莉金（binlijin） 
I like to use the former, heap occupancy, so we not need to worry about the
OOM and FullGc，and change configuration to adapted to new policy.

2017-07-06 14:03 GMT+08:00 Stack :

> On Wed, Jul 5, 2017 at 9:59 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> >
> > >>Sounds like we should be doing the former, heap occupancy
> > Stack, so do you mean we need to roll back this new change in trunk? The
> > background is https://issues.apache.org/jira/browse/HBASE-16747.
> >
> >
> I remember that issue. It seems good to me (as it did then) where we have
> the global tracking in RS of all data and overhead so we shouldn't OOME and
> we keep accounting of overhead and data distinct because now data can be
> onheap or offheap.
>
> We shouldn't be doing blocking updates -- not when there is probably loads
> of memory still available -- but that is a different (critical) issue.
> Sounds like current configs can 'surprise' -- see Yu Li note -- given the
> new accounting.
>
> Looks like I need to read HBASE-18294
>  to figure what the
> pivot/problem w/ the new policy is.
>
> Thanks,
> St.Ack
>
>
>
>
>
> > Regards
> > Ram
> >
> >
> > On Thu, Jul 6, 2017 at 8:40 AM, Yu Li  wrote:
> >
> > > We've also observed more blocking updates happening with the new policy
> > > (flush decision made on data size), but could work-around it by
> reducing
> > > the hbase.hregion.memstore.flush.size setting. The advantage of
> current
> > > policy is we could control the flushed file size more accurately, but
> > > meanwhile losing some "compatibility" (requires configuration updating
> > > during rolling upgrade).
> > >
> > > I'm not sure whether we should rollback, but if stick on current policy
> > > there should be more documents, metrics (monitoring heap/data occupancy
> > > separately) and log message refinements, etc. Attaching some of the
> logs
> > we
> > > observed, which is pretty confusing w/o knowing the details of
> > > implementation:
> > >
> > > 2017-07-03 16:11:54,724 INFO
> > >  [B.defaultRpcServer.handler=182,queue=11,port=16020]
> > > regionserver.MemStoreFlusher: Blocking updates on
> > > hadoop0528.et2.tbsite.net,16020,1497336978160:
> > > global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> > > 2017-07-03 16:11:54,754 INFO
> > >  [B.defaultRpcServer.handler=186,queue=15,port=16020]
> > > regionserver.MemStoreFlusher: Blocking updates on
> > > hadoop0528.et2.tbsite.net,16020,1497336978160:
> > > global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> > > 2017-07-03 16:11:57,571 INFO  [MemStoreFlusher.0]
> > > regionserver.MemStoreFlusher: Flush of region
> > > mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e1649
> > 2a.
> > > due to global heap pressure. Total Memstore size=3.2 G, Region memstore
> > > size=331.4 M
> > > 2017-07-03 16:11:57,571 WARN
> > >  [B.defaultRpcServer.handler=49,queue=11,port=16020]
> > > regionserver.MemStoreFlusher: Memstore is above high water mark and
> block
> > > 2892ms
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 6 July 2017 at 00:56, Stack  wrote:
> > >
> > > > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel
> > >  > > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > > I opened a new Jira https://issues.apache.org/
> > jira/browse/HBASE-18294
> > > to
> > > > > discuss this question.
> > > > > Flush decisions are taken at the region level and also at the
> region
> > > > > server level - there is the question of when to trigger a flush and
> > > then
> > > > > which region/store to flush.Regions track both their data size
> > > (key-value
> > > > > size only) and their total heap occupancy (including index and
> > > additional
> > > > > metadata).One option (which was the past policy) is to trigger
> > flushes
> > > > and
> > > > > choose flush subjects based on regions heap size - this gives a
> > better
> > > > > estimation for sysadmin of how many regions can a RS carry.Another
> > > option
> > > > > (which is the current policy) is to look at the data size - this
> > gives
> > > a
> > > > > better estimation of the size of the files that are created by the
> > > flush.
> > > > >
> > > >
> > > >
> > > > Sounds like we should be doing the former, heap occupancy. An
> > > > OutOfMemoryException puts a nail in any benefit other accountings
> might
> > > > have.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > > I see this is as critical to HBase performance and usability,
> namely
> > > > > meeting the user expectation from the system, hence I would like to
> > > hear
> > > > as
> > > > > many voices as possible.Please join the discussion in the Jira and
> > let
> > > us
> > > > > know what you think.
> > > > > Thanks,Eshcar
> > > > >
> > > > >
> > > >
> > >
> >
>



-- 
*Best Regards,*
 lijin bin

Re: [DISCUSS] Should flush decisions be made based on data size (key-value only) or based on heap size (including metadata overhead)?

2017-07-06 Thread Stack 
On Wed, Jul 5, 2017 at 9:59 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

>
> >>Sounds like we should be doing the former, heap occupancy
> Stack, so do you mean we need to roll back this new change in trunk? The
> background is https://issues.apache.org/jira/browse/HBASE-16747.
>
>
I remember that issue. It seems good to me (as it did then) where we have
the global tracking in RS of all data and overhead so we shouldn't OOME and
we keep accounting of overhead and data distinct because now data can be
onheap or offheap.

We shouldn't be doing blocking updates -- not when there is probably loads
of memory still available -- but that is a different (critical) issue.
Sounds like current configs can 'surprise' -- see Yu Li note -- given the
new accounting.

Looks like I need to read HBASE-18294
 to figure what the
pivot/problem w/ the new policy is.

Thanks,
St.Ack





> Regards
> Ram
>
>
> On Thu, Jul 6, 2017 at 8:40 AM, Yu Li  wrote:
>
> > We've also observed more blocking updates happening with the new policy
> > (flush decision made on data size), but could work-around it by reducing
> > the hbase.hregion.memstore.flush.size setting. The advantage of current
> > policy is we could control the flushed file size more accurately, but
> > meanwhile losing some "compatibility" (requires configuration updating
> > during rolling upgrade).
> >
> > I'm not sure whether we should rollback, but if stick on current policy
> > there should be more documents, metrics (monitoring heap/data occupancy
> > separately) and log message refinements, etc. Attaching some of the logs
> we
> > observed, which is pretty confusing w/o knowing the details of
> > implementation:
> >
> > 2017-07-03 16:11:54,724 INFO
> >  [B.defaultRpcServer.handler=182,queue=11,port=16020]
> > regionserver.MemStoreFlusher: Blocking updates on
> > hadoop0528.et2.tbsite.net,16020,1497336978160:
> > global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> > 2017-07-03 16:11:54,754 INFO
> >  [B.defaultRpcServer.handler=186,queue=15,port=16020]
> > regionserver.MemStoreFlusher: Blocking updates on
> > hadoop0528.et2.tbsite.net,16020,1497336978160:
> > global memstore heapsize 7.2 G is >= than blocking 7.2 G size
> > 2017-07-03 16:11:57,571 INFO  [MemStoreFlusher.0]
> > regionserver.MemStoreFlusher: Flush of region
> > mainv7_main_result_c,1496,1499062935573.02adfa7cbdc606dce5b79a516e1649
> 2a.
> > due to global heap pressure. Total Memstore size=3.2 G, Region memstore
> > size=331.4 M
> > 2017-07-03 16:11:57,571 WARN
> >  [B.defaultRpcServer.handler=49,queue=11,port=16020]
> > regionserver.MemStoreFlusher: Memstore is above high water mark and block
> > 2892ms
> >
> > Best Regards,
> > Yu
> >
> > On 6 July 2017 at 00:56, Stack  wrote:
> >
> > > On Wed, Jul 5, 2017 at 6:30 AM, Eshcar Hillel
> >  > > >
> > > wrote:
> > >
> > > > Hi All,
> > > > I opened a new Jira https://issues.apache.org/
> jira/browse/HBASE-18294
> > to
> > > > discuss this question.
> > > > Flush decisions are taken at the region level and also at the region
> > > > server level - there is the question of when to trigger a flush and
> > then
> > > > which region/store to flush.Regions track both their data size
> > (key-value
> > > > size only) and their total heap occupancy (including index and
> > additional
> > > > metadata).One option (which was the past policy) is to trigger
> flushes
> > > and
> > > > choose flush subjects based on regions heap size - this gives a
> better
> > > > estimation for sysadmin of how many regions can a RS carry.Another
> > option
> > > > (which is the current policy) is to look at the data size - this
> gives
> > a
> > > > better estimation of the size of the files that are created by the
> > flush.
> > > >
> > >
> > >
> > > Sounds like we should be doing the former, heap occupancy. An
> > > OutOfMemoryException puts a nail in any benefit other accountings might
> > > have.
> > >
> > > St.Ack
> > >
> > >
> > >
> > > > I see this is as critical to HBase performance and usability, namely
> > > > meeting the user expectation from the system, hence I would like to
> > hear
> > > as
> > > > many voices as possible.Please join the discussion in the Jira and
> let
> > us
> > > > know what you think.
> > > > Thanks,Eshcar
> > > >
> > > >
> > >
> >
>