date:20150830

Sean Busbey created HBASE-14338:
---

 Summary: License notification misspells 'Asciidoctor'
 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Priority: Minor


our License file contains 'asciidoctor' but with three i

{quote}
This project bundles a derivative of portions of the 'Asciiidoctor' project
under the terms of the MIT license.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14340) Add second bulk load option to Spark Bulk Load to send puts as the value

Ted Malaska created HBASE-14340:
---

 Summary: Add second bulk load option to Spark Bulk Load to send 
puts as the value
 Key: HBASE-14340
 URL: https://issues.apache.org/jira/browse/HBASE-14340
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor


The initial bulk load option for Spark bulk load sends values over one by one 
through the shuffle.  This is the similar to how the original MR bulk load 
worked.

How ever the MR bulk loader have more then one bulk load option.  There is a 
second option that allows for all the Column Families, Qualifiers, and Values 
or a row to be combined in the map side.

This only works if the row is not super wide.

But if the row is not super wide this method of sending values through the 
shuffle will reduce the data and work the shuffle has to deal with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14339) HBase Bulk Load and super wide rows

Ted Malaska created HBASE-14339:
---

 Summary: HBase Bulk Load and super wide rows
 Key: HBASE-14339
 URL: https://issues.apache.org/jira/browse/HBASE-14339
 Project: HBase
  Issue Type: Bug
Reporter: Ted Malaska
Priority: Minor


This may not be a huge issues but it does come up.  If the number of columns in 
a row are to many then KeyValueSortReducer will blow up with a out of memory 
exception, because it uses a TreeMap to sort the columns with in the memory of 
the reducer.

A solution would be to add the column family and qualifier to the key so the 
shuffle would handle the sort.

The partitioner would only partition on the rowKey but ordering would apply to 
the RowKey, Column Family, and Column Qualifier.

Look at the Spark Bulk load as an example.  HBASE-14150  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module


[ 
https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721718#comment-14721718
 ] 

Ted Malaska commented on HBASE-14181:
-

I just rechecked this patch and I had no trouble compiling and running the test.

What am I missing?

I did the follwing
1. mvn clean package -DskipTests=true in the root directory
2. mvn -Dtest=NoUnitTests clean verify in the hbase-spark folder

Thanks for the help

 Add Spark DataFrame DataSource to HBase-Spark Module
 

 Key: HBASE-14181
 URL: https://issues.apache.org/jira/browse/HBASE-14181
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
 Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch, 
 HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-14181.5.patch, 
 HBASE-14181.6.patch, HBASE-14181.7.patch, HBASE-14181.8.patch


 Build a RelationProvider for HBase-Spark Module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module


[ 
https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721725#comment-14721725
 ] 

Sean Busbey commented on HBASE-14181:
-

it's HBASE-14337. specifically, when building with Hadoop 2.4 we're missing a 
license for some transitive dependency.

 Add Spark DataFrame DataSource to HBase-Spark Module
 

 Key: HBASE-14181
 URL: https://issues.apache.org/jira/browse/HBASE-14181
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
 Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch, 
 HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-14181.5.patch, 
 HBASE-14181.6.patch, HBASE-14181.7.patch, HBASE-14181.8.patch


 Build a RelationProvider for HBase-Spark Module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14339) HBase Bulk Load and super wide rows

2015-08-30 Thread Jean-Marc Spaggiari (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721748#comment-14721748
 ] 

Jean-Marc Spaggiari commented on HBASE-14339:
-

If I'm not mistaken someone is already fixing that in another JIRA. I will try 
to find it tomorrow...

 HBase Bulk Load and super wide rows
 ---

 Key: HBASE-14339
 URL: https://issues.apache.org/jira/browse/HBASE-14339
 Project: HBase
  Issue Type: Bug
Reporter: Ted Malaska
Priority: Minor

 This may not be a huge issues but it does come up.  If the number of columns 
 in a row are to many then KeyValueSortReducer will blow up with a out of 
 memory exception, because it uses a TreeMap to sort the columns with in the 
 memory of the reducer.
 A solution would be to add the column family and qualifier to the key so the 
 shuffle would handle the sort.
 The partitioner would only partition on the rowKey but ordering would apply 
 to the RowKey, Column Family, and Column Qualifier.
 Look at the Spark Bulk load as an example.  HBASE-14150  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table


[ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721539#comment-14721539
 ] 

Hadoop QA commented on HBASE-14332:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12753184/HBASE-14332_v2.patch
  against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
  ATTACHMENT ID: 12753184

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:red}-1 javac{color}.  The patch appears to cause mvn compile goal to 
fail with Hadoop version 2.4.0.

Compilation errors resume:
[ERROR] Error invoking method 'get(java.lang.Integer)' in 
java.util.ArrayList at META-INF/LICENSE.vm[line 1619, column 22]
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on 
project hbase-assembly: Error rendering velocity resource. Error invoking 
method 'get(java.lang.Integer)' in java.util.ArrayList at 
META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: 0, 
Size: 0 - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hbase-assembly


Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15328//console

This message is automatically generated.

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table


[ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721537#comment-14721537
 ] 

Nick.han commented on HBASE-14332:
--

I recreated a patch which fixed line length check error,and I guess line 
lengths error lead to check style error,so I fix line length error,I fix check 
style error.  In addition the original test case have already covered the 
change I did,so I didn't add any test case, I know there is more thing I need 
to do,but I'm not familiar with the Hadoop QA'test,so,any suggestion for me to 
make the patch qualified?Thank you very much.

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12988) [Replication]Parallel apply edits across regions

2015-08-30 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721564#comment-14721564
 ] 

Elliott Clark commented on HBASE-12988:
---

+1 from me.

 [Replication]Parallel apply edits across regions
 

 Key: HBASE-12988
 URL: https://issues.apache.org/jira/browse/HBASE-12988
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Reporter: hongyu bi
Assignee: Lars Hofhansl
 Attachments: 12988-v2.txt, 12988-v3.txt, 12988-v4.txt, 12988-v5.txt, 
 12988.txt, HBASE-12988-0.98.patch, ParallelReplication-v2.txt


 we can apply  edits to slave cluster in parallel on table-level to speed up 
 replication .
 update : per conversation blow , it's better to apply edits on row-level in 
 parallel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table

2015-08-30 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721573#comment-14721573
 ] 

Ted Yu commented on HBASE-14332:


{code}
80 * @return null if successed or table state if failed
{code}
successed: typo
{code}
297   TableState.State state = tsm.getTableState(tableName);
298   //if 
(!tsm.getTableState(tableName).equals(TableState.State.ENABLED)) {
...
317   TableState.State state = tsm.getTableState(tableName);
318   //if 
(!tsm.getTableState(tableName).equals(TableState.State.DISABLED)) {
{code}
Remove commented out code above.

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction

2015-08-30 Thread Edward Bortnikov (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721512#comment-14721512
]

Edward Bortnikov commented on HBASE-13408:
--

We did our homework to address [~anoop.hbase]'s and [~mbertozzi]'s concerns.
In-memory HFile's are absolutely possible to implement. The challenge with the
current implementation is that the StoreFile implementation is too tightly
coupled with HDFS whereas it could be associated with byte stream just as well.
Most of the code is FS-independent, however some accurate refactoring and a
couple of new abstractions would be required. Our concern is that this code is
already a bulk, and further expanding it only increases the risk. Would it make
sense do the following: (1) start reviewing and checking-in the existing code,
either in one bulk or piecemeal, and (2) in parallel, design, implement and
evaluate in-memory HFile's, either in a separate jira or as subtask in this
jira. This could be just one more patch in the series.

HBase In-Memory Memstore Compaction
---

Key: HBASE-13408
URL: https://issues.apache.org/jira/browse/HBASE-13408
Project: HBase
Issue Type: New Feature
Reporter: Eshcar Hillel
Attachments: HBASE-13408-trunk-v01.patch,
HBASE-13408-trunk-v02.patch,
HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf,
HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf,
InMemoryMemstoreCompactionEvaluationResults.pdf,
InMemoryMemstoreCompactionScansEvaluationResults.pdf

A store unit holds a column family in a region, where the memstore is its
in-memory component. The memstore absorbs all updates to the store; from time
to time these updates are flushed to a file on disk, where they are
compacted. Unlike disk components, the memstore is not compacted until it is
written to the filesystem and optionally to block-cache. This may result in
underutilization of the memory due to duplicate entries per row, for example,
when hot data is continuously updated.
Generally, the faster the data is accumulated in memory, more flushes are
triggered, the data sinks to disk more frequently, slowing down retrieval of
data, even if very recent.
In high-churn workloads, compacting the memstore can help maintain the data
in memory, and thereby speed up data retrieval.
We suggest a new compacted memstore with the following principles:
1.The data is kept in memory for as long as possible
2.Memstore data is either compacted or in process of being compacted
3.Allow a panic mode, which may interrupt an in-progress compaction and
force a flush of part of the memstore.
We suggest applying this optimization only to in-memory column families.
A design document is attached.
This feature was previously discussed in HBASE-5311.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table


 [ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick.han updated HBASE-14332:
-
Attachment: HBASE-14332_v2.patch

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14309) Allow load balancer to operate when there is region in transition by adding force flag

2015-08-30 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14309:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 Allow load balancer to operate when there is region in transition by adding 
 force flag
 --

 Key: HBASE-14309
 URL: https://issues.apache.org/jira/browse/HBASE-14309
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 2.0.0, 1.3.0

 Attachments: 14309-branch-1.1.txt, 14309-v1.txt, 14309-v2.txt, 
 14309-v3.txt, 14309-v4.txt, 14309-v5-branch-1.txt, 14309-v5.txt, 
 14309-v5.txt, 14309-v6.txt, 14309-v7-branch-1.txt, 14309-v7.txt


 This issue adds boolean parameter, force, to 'balancer' command so that admin 
 can force region balancing even when there is region in transition - assuming 
 RIT being transient.
 This enhancement was requested by some customer.
 The assumption of this change is that the operator has run hbck and has a 
 reasonable idea why regions are stuck in transition before using the force 
 flag.
 There was a recent event at the customer where a cluster ended up with a 
 small number of regionservers hosting most of the regions on the cluster (one 
 regionserver had 50% of the roughly 20,000 regions). The balancer couldn't be 
 run due to the small number of regions that were stuck in transition. The 
 admin ended up killing the regionservers so that reassignment would yield a 
 more equitable distribution of the regions.
 On a different cluster, there was a single store file that had corrupt HDFS 
 blocks (the SSDs on the cluster were known to lose data). However, since this 
 single region (out of 10s of 1000s of regions on this cluster) was stuck in 
 transition, the balancer couldn't run.
 While the state keeping in HBase isn't so good yet that the admin can kick 
 off the balancer automatically in such scenarios knowing when it is safe to 
 do so and when it is not, having this option available for the operator to 
 use as he / she sees fit seems prudent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14341) branch-1.1 source assembly contains spurious hbase-shaded-client and hbase-shaded-server modules

Sean Busbey created HBASE-14341:
---

 Summary: branch-1.1 source assembly contains spurious 
hbase-shaded-client and hbase-shaded-server modules
 Key: HBASE-14341
 URL: https://issues.apache.org/jira/browse/HBASE-14341
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 1.1.2
Reporter: Sean Busbey


if you build a source assembly according to the book:

{quote}
mvn clean install -DskipTests assembly:single 
-Dassembly.file=hbase-assembly/src/main/assembly/src.xml -Prelease
{quote}

Then the resultant artifact has an extra set of the shaded modules at the top 
level (in addition to the ones in the hbase-shaded module)

{code}
$ ls -lah hbase-1.1.2/
total 608
drwxr-xr-x  32 busbey  staff   1.1K Aug 30 17:14 .
drwxr-xr-x   3 busbey  staff   102B Aug 30 17:14 ..
-rw-r--r--   1 busbey  staff   162K Aug 30 16:42 CHANGES.txt
-rw-r--r--   1 busbey  staff36K Aug 30 16:15 LICENSE.txt
-rw-r--r--   1 busbey  staff   1.5K Aug 30 16:15 NOTICE.txt
-rw-r--r--   1 busbey  staff   1.4K Aug 30 16:15 README.txt
drwxr-xr-x  31 busbey  staff   1.0K Aug 30 16:42 bin
drwxr-xr-x   9 busbey  staff   306B Aug 30 16:42 conf
drwxr-xr-x  24 busbey  staff   816B Aug 30 16:42 dev-support
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-annotations
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-assembly
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-checkstyle
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-client
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-common
drwxr-xr-x   5 busbey  staff   170B Aug 30 16:58 hbase-examples
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-hadoop-compat
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-hadoop2-compat
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-it
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-prefix-tree
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-procedure
drwxr-xr-x   5 busbey  staff   170B Aug 30 16:56 hbase-protocol
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-resource-bundle
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-rest
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-server
drwxr-xr-x   5 busbey  staff   170B Aug 30 16:42 hbase-shaded
drwxr-xr-x   3 busbey  staff   102B Aug 30 16:42 hbase-shaded-client
drwxr-xr-x   3 busbey  staff   102B Aug 30 16:42 hbase-shaded-server
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-shell
drwxr-xr-x   3 busbey  staff   102B Aug 30 16:57 hbase-testing-util
drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-thrift
-rw-r--r--   1 busbey  staff94K Aug 30 16:42 pom.xml
drwxr-xr-x   3 busbey  staff   102B Aug 30 16:15 src
$ diff -r hbase-1.1.2/hbase-shaded-client 
hbase-1.1.2/hbase-shaded/hbase-shaded-client
Only in hbase-1.1.2/hbase-shaded/hbase-shaded-client: target
$ diff -r hbase-1.1.2/hbase-shaded-server 
hbase-1.1.2/hbase-shaded/hbase-shaded-server
Only in hbase-1.1.2/hbase-shaded/hbase-shaded-server: target
{code}

they're the same as the correct ones and they don't build by default since the 
top level pom doesn't mention them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14338) License notification misspells 'Asciidoctor'

2015-08-30 Thread Lars Francke (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721762#comment-14721762
 ] 

Lars Francke commented on HBASE-14338:
--

Patch applies in master and branch-1 as far as I can tell

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Ted Malaska
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14321) branch-1.1 src artifact includes shaded target dirs


[ 
https://issues.apache.org/jira/browse/HBASE-14321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721763#comment-14721763
 ] 

Sean Busbey commented on HBASE-14321:
-

I hit this while following the ref guide's instructions for building the source 
assembly:

{code}
$ mvn clean install -DskipTests assembly:single 
-Dassembly.file=hbase-assembly/src/main/assembly/src.xml -Prelease
{code}

 branch-1.1 src artifact includes shaded target dirs
 ---

 Key: HBASE-14321
 URL: https://issues.apache.org/jira/browse/HBASE-14321
 Project: HBase
  Issue Type: Task
  Components: build
Affects Versions: 1.1.2
Reporter: Nick Dimiduk
Priority: Blocker

 Fixed the make_rc script, now I find in the src binary.
 {noformat}
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-archiver/
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-archiver/
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/pom.xml
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/.plxarc
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/LICENSE
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/NOTICE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/dependency-reduced-pom.xml
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-1.1.2.jar
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-archiver/pom.properties
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/NOTICE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/original-hbase-shaded-client-1.1.2.jar
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/rat.txt
 x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/supplemental-models.xml
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/pom.xml
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/.plxarc
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/LICENSE
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/NOTICE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/dependency-reduced-pom.xml
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/hbase-shaded-server-1.1.2.jar
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-archiver/pom.properties
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/LICENSE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/NOTICE
 x 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/original-hbase-shaded-server-1.1.2.jar
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/rat.txt
 x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/supplemental-models.xml
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14338) License notification misspells 'Asciidoctor'

2015-08-30 Thread Lars Francke (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HBASE-14338:
-
Attachment: HBASE-14338.patch

Sorry Ted had something already but forgot to assign...

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Ted Malaska
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12187) Review in source the paper Simple Testing Can Prevent Most Critical Failures

2015-08-30 Thread Ding Yuan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721858#comment-14721858
 ] 

Ding Yuan commented on HBASE-12187:
---

Hi [~busbey], i don't think so. I remember I sent error-prone's developers the 
patches last year via google group, but I can no longer find those discussion 
threads (seems they migrated the whole thing to github). Should I send the 
patch to the error-prone devs again or you guys are hosting error-prone 
repository locally?

 Review in source the paper Simple Testing Can Prevent Most Critical Failures
 --

 Key: HBASE-12187
 URL: https://issues.apache.org/jira/browse/HBASE-12187
 Project: HBase
  Issue Type: Bug
Reporter: stack
Priority: Critical
 Attachments: HBASE-12187.patch, abortInOvercatch.warnings.txt, 
 emptyCatch.warnings.txt, todoInCatch.warnings.txt


 Review the helpful paper 
 https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
 It describes 'catastrophic failures', especially issues where exceptions are 
 thrown but not properly handled.  Their static analysis tool Aspirator turns 
 up a bunch of the obvious offenders (Lets add to test-patch.sh alongside 
 findbugs?).  This issue is about going through code base making sub-issues to 
 root out these and others (Don't we have the test described in figure #6 
 already? I thought we did?  If we don't, need to add).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14158) Add documentation for Initial Release for HBase-Spark Module integration


 [ 
https://issues.apache.org/jira/browse/HBASE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska updated HBASE-14158:

Attachment: HBASE-14158.5.patch

This has a bunch of fixes.

1. All of Sean B's comments
2. A bunch of Asciidoc fixes
3. Fixed all trailing white spaces
4. Tried to get all lines under 100 characters

 Add documentation for Initial Release for HBase-Spark Module integration 
 -

 Key: HBASE-14158
 URL: https://issues.apache.org/jira/browse/HBASE-14158
 Project: HBase
  Issue Type: Improvement
  Components: documentation, spark
Reporter: Ted Malaska
Assignee: Ted Malaska
 Fix For: 2.0.0

 Attachments: HBASE-14158.1.patch, HBASE-14158.2.patch, 
 HBASE-14158.5.patch


 Add documentation for Initial Release for HBase-Spark Module integration 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module


[ 
https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721756#comment-14721756
 ] 

Ted Malaska commented on HBASE-14181:
-

What should I do?

Let me know what else is needed on this one.  This and the documentation are 
the two last big ones.  

 Add Spark DataFrame DataSource to HBase-Spark Module
 

 Key: HBASE-14181
 URL: https://issues.apache.org/jira/browse/HBASE-14181
 Project: HBase
  Issue Type: New Feature
  Components: spark
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
 Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch, 
 HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-14181.5.patch, 
 HBASE-14181.6.patch, HBASE-14181.7.patch, HBASE-14181.8.patch


 Build a RelationProvider for HBase-Spark Module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14338) License notification misspells 'Asciidoctor'


[ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721779#comment-14721779
 ] 

Hadoop QA commented on HBASE-14338:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12753216/HBASE-14338.patch
  against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
  ATTACHMENT ID: 12753216

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation, build,
or dev-support patch that doesn't require tests.

{color:red}-1 javac{color}.  The patch appears to cause mvn compile goal to 
fail with Hadoop version 2.4.0.

Compilation errors resume:
[ERROR] Error invoking method 'get(java.lang.Integer)' in 
java.util.ArrayList at META-INF/LICENSE.vm[line 1619, column 22]
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on 
project hbase-assembly: Error rendering velocity resource. Error invoking 
method 'get(java.lang.Integer)' in java.util.ArrayList at 
META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: 0, 
Size: 0 - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hbase-assembly


Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15330//console

This message is automatically generated.

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Lars Francke
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs

2015-08-30 Thread Heng Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14227:
--
Attachment: (was: HBASE-14227_v5.patch)

 Fold special cased MOB APIs into existing APIs
 --

 Key: HBASE-14227
 URL: https://issues.apache.org/jira/browse/HBASE-14227
 Project: HBase
  Issue Type: Task
  Components: mob
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Heng Chen
Priority: Blocker
 Fix For: 2.0.0

 Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, 
 HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch


 There are a number of APIs that came in with MOB that are not new actions for 
 HBase, simply new actions for a MOB implementation:
 - compactMob
 - compactMobs
 - majorCompactMob
 - majorCompactMobs
 - getMobCompactionState
 And in HBaseAdmin:
 - validateMobColumnFamily
 Remove these special cases from the Admin API where possible by folding them 
 into existing APIs.
 We definitely don't need one method for a singleton and another for 
 collections.
 Ideally we will not have any APIs named *Mob when finished, whether MOBs are 
 in use on a table or not should be largely an internal detail. Exposing as 
 schema option would be fine, this conforms to existing practice for other 
 features.
 Marking critical because I think removing the *Mob special cased APIs should 
 be a precondition for release of this feature either in 2.0 or as a backport.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs

2015-08-30 Thread Heng Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14227:
--
Attachment: HBASE-14227_v5.patch

retry again.

 Fold special cased MOB APIs into existing APIs
 --

 Key: HBASE-14227
 URL: https://issues.apache.org/jira/browse/HBASE-14227
 Project: HBase
  Issue Type: Task
  Components: mob
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Heng Chen
Priority: Blocker
 Fix For: 2.0.0

 Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, 
 HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, 
 HBASE-14227_v5.patch


 There are a number of APIs that came in with MOB that are not new actions for 
 HBase, simply new actions for a MOB implementation:
 - compactMob
 - compactMobs
 - majorCompactMob
 - majorCompactMobs
 - getMobCompactionState
 And in HBaseAdmin:
 - validateMobColumnFamily
 Remove these special cases from the Admin API where possible by folding them 
 into existing APIs.
 We definitely don't need one method for a singleton and another for 
 collections.
 Ideally we will not have any APIs named *Mob when finished, whether MOBs are 
 in use on a table or not should be largely an internal detail. Exposing as 
 schema option would be fine, this conforms to existing practice for other 
 features.
 Marking critical because I think removing the *Mob special cased APIs should 
 be a precondition for release of this feature either in 2.0 or as a backport.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14338) License notification misspells 'Asciidoctor'


 [ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska updated HBASE-14338:

Assignee: Lars Francke  (was: Ted Malaska)

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Lars Francke
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14341) branch-1.1 source assembly contains spurious hbase-shaded-client and hbase-shaded-server modules


[ 
https://issues.apache.org/jira/browse/HBASE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721767#comment-14721767
 ] 

Sean Busbey commented on HBASE-14341:
-

these same extra directories show up in 1.1.2RC2

 branch-1.1 source assembly contains spurious hbase-shaded-client and 
 hbase-shaded-server modules
 

 Key: HBASE-14341
 URL: https://issues.apache.org/jira/browse/HBASE-14341
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 1.1.2
Reporter: Sean Busbey

 if you build a source assembly according to the book:
 {quote}
 mvn clean install -DskipTests assembly:single 
 -Dassembly.file=hbase-assembly/src/main/assembly/src.xml -Prelease
 {quote}
 Then the resultant artifact has an extra set of the shaded modules at the top 
 level (in addition to the ones in the hbase-shaded module)
 {code}
 $ ls -lah hbase-1.1.2/
 total 608
 drwxr-xr-x  32 busbey  staff   1.1K Aug 30 17:14 .
 drwxr-xr-x   3 busbey  staff   102B Aug 30 17:14 ..
 -rw-r--r--   1 busbey  staff   162K Aug 30 16:42 CHANGES.txt
 -rw-r--r--   1 busbey  staff36K Aug 30 16:15 LICENSE.txt
 -rw-r--r--   1 busbey  staff   1.5K Aug 30 16:15 NOTICE.txt
 -rw-r--r--   1 busbey  staff   1.4K Aug 30 16:15 README.txt
 drwxr-xr-x  31 busbey  staff   1.0K Aug 30 16:42 bin
 drwxr-xr-x   9 busbey  staff   306B Aug 30 16:42 conf
 drwxr-xr-x  24 busbey  staff   816B Aug 30 16:42 dev-support
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-annotations
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-assembly
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-checkstyle
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-client
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-common
 drwxr-xr-x   5 busbey  staff   170B Aug 30 16:58 hbase-examples
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-hadoop-compat
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-hadoop2-compat
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-it
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-prefix-tree
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-procedure
 drwxr-xr-x   5 busbey  staff   170B Aug 30 16:56 hbase-protocol
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:56 hbase-resource-bundle
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-rest
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-server
 drwxr-xr-x   5 busbey  staff   170B Aug 30 16:42 hbase-shaded
 drwxr-xr-x   3 busbey  staff   102B Aug 30 16:42 hbase-shaded-client
 drwxr-xr-x   3 busbey  staff   102B Aug 30 16:42 hbase-shaded-server
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:58 hbase-shell
 drwxr-xr-x   3 busbey  staff   102B Aug 30 16:57 hbase-testing-util
 drwxr-xr-x   4 busbey  staff   136B Aug 30 16:57 hbase-thrift
 -rw-r--r--   1 busbey  staff94K Aug 30 16:42 pom.xml
 drwxr-xr-x   3 busbey  staff   102B Aug 30 16:15 src
 $ diff -r hbase-1.1.2/hbase-shaded-client 
 hbase-1.1.2/hbase-shaded/hbase-shaded-client
 Only in hbase-1.1.2/hbase-shaded/hbase-shaded-client: target
 $ diff -r hbase-1.1.2/hbase-shaded-server 
 hbase-1.1.2/hbase-shaded/hbase-shaded-server
 Only in hbase-1.1.2/hbase-shaded/hbase-shaded-server: target
 {code}
 they're the same as the correct ones and they don't build by default since 
 the top level pom doesn't mention them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14338) License notification misspells 'Asciidoctor'


[ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721765#comment-14721765
 ] 

Ted Malaska commented on HBASE-14338:
-

No problem.  Thanks

I changed the Assignee.

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Lars Francke
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14158) Add documentation for Initial Release for HBase-Spark Module integration


[ 
https://issues.apache.org/jira/browse/HBASE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721758#comment-14721758
 ] 

Hadoop QA commented on HBASE-14158:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12753215/HBASE-14158.5.patch
  against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
  ATTACHMENT ID: 12753215

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:red}-1 javac{color}.  The patch appears to cause mvn compile goal to 
fail with Hadoop version 2.4.0.

Compilation errors resume:
[ERROR] Error invoking method 'get(java.lang.Integer)' in 
java.util.ArrayList at META-INF/LICENSE.vm[line 1619, column 22]
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on 
project hbase-assembly: Error rendering velocity resource. Error invoking 
method 'get(java.lang.Integer)' in java.util.ArrayList at 
META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: 0, 
Size: 0 - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hbase-assembly


Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15329//console

This message is automatically generated.

 Add documentation for Initial Release for HBase-Spark Module integration 
 -

 Key: HBASE-14158
 URL: https://issues.apache.org/jira/browse/HBASE-14158
 Project: HBase
  Issue Type: Improvement
  Components: documentation, spark
Reporter: Ted Malaska
Assignee: Ted Malaska
 Fix For: 2.0.0

 Attachments: HBASE-14158.1.patch, HBASE-14158.2.patch, 
 HBASE-14158.5.patch


 Add documentation for Initial Release for HBase-Spark Module integration 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14338) License notification misspells 'Asciidoctor'


 [ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-14338:

Status: Patch Available  (was: Open)

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Lars Francke
Priority: Minor
 Attachments: HBASE-14338.patch


 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14338) License notification misspells 'Asciidoctor'


 [ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska reassigned HBASE-14338:
---

Assignee: Ted Malaska

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Ted Malaska
Priority: Minor

 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14338) License notification misspells 'Asciidoctor'


[ 
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721760#comment-14721760
 ] 

Ted Malaska commented on HBASE-14338:
-

Will have something shortly

 License notification misspells 'Asciidoctor'
 

 Key: HBASE-14338
 URL: https://issues.apache.org/jira/browse/HBASE-14338
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Ted Malaska
Priority: Minor

 our License file contains 'asciidoctor' but with three i
 {quote}
 This project bundles a derivative of portions of the 'Asciiidoctor' project
 under the terms of the MIT license.
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs


[ 
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721933#comment-14721933
 ] 

Hadoop QA commented on HBASE-14227:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12753222/HBASE-14227_v5.patch
  against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
  ATTACHMENT ID: 12753222

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified tests.

{color:red}-1 javac{color}.  The patch appears to cause mvn compile goal to 
fail with Hadoop version 2.4.0.

Compilation errors resume:
[ERROR] Error invoking method 'get(java.lang.Integer)' in 
java.util.ArrayList at META-INF/LICENSE.vm[line 1619, column 22]
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on 
project hbase-assembly: Error rendering velocity resource. Error invoking 
method 'get(java.lang.Integer)' in java.util.ArrayList at 
META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: 0, 
Size: 0 - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hbase-assembly


Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15331//console

This message is automatically generated.

 Fold special cased MOB APIs into existing APIs
 --

 Key: HBASE-14227
 URL: https://issues.apache.org/jira/browse/HBASE-14227
 Project: HBase
  Issue Type: Task
  Components: mob
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Heng Chen
Priority: Blocker
 Fix For: 2.0.0

 Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, 
 HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, 
 HBASE-14227_v5.patch


 There are a number of APIs that came in with MOB that are not new actions for 
 HBase, simply new actions for a MOB implementation:
 - compactMob
 - compactMobs
 - majorCompactMob
 - majorCompactMobs
 - getMobCompactionState
 And in HBaseAdmin:
 - validateMobColumnFamily
 Remove these special cases from the Admin API where possible by folding them 
 into existing APIs.
 We definitely don't need one method for a singleton and another for 
 collections.
 Ideally we will not have any APIs named *Mob when finished, whether MOBs are 
 in use on a table or not should be largely an internal detail. Exposing as 
 schema option would be fine, this conforms to existing practice for other 
 features.
 Marking critical because I think removing the *Mob special cased APIs should 
 be a precondition for release of this feature either in 2.0 or as a backport.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13161) ITBLL needs love; verify spins w/o end at 10B rows

2015-08-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14722088#comment-14722088
 ] 

stack commented on HBASE-13161:
---

Here is the commands I ran:

${HOME}/hadoop/bin/hadoop --config ${HOME}/conf_hadoop 
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey 
serverKilling Generator 40 25000 g.$date

${HOME}/hadoop/bin/hadoop --config ${HOME}/conf_hadoop 
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList Verify v.$date 5

 ITBLL needs love; verify spins w/o end at 10B rows
 --

 Key: HBASE-13161
 URL: https://issues.apache.org/jira/browse/HBASE-13161
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Reporter: stack
Assignee: stack
 Fix For: 1.2.0


 Making a note of my recent experience with ITBLL; here is stuff that needs 
 fixing.
 + I was able to run generate for about 24 hours with monkey going and loaded 
 10B rows. When I ran the verify (a few times), it got stuck at about 90odd 
 percent and never made progress beyond that (this was about 20 hours in 
 IIRC). I spent no time trying to figure why (ran out of cluster and time).
 + Verify at least takes too long. Need to make it run faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14331) a single callQueue related improvements

2015-08-30 Thread Hiroshi Ikeda (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14722832#comment-14722832
 ] 

Hiroshi Ikeda commented on HBASE-14331:
---

{{BoundedConcurrentLinkedQueue}} is a class which just extends 
{{ConcurrentLinkedQueue}} and adds the boundary (but it seems to have bugs 
about {{Iterator.remove}} and methods internally using the 
{{Iterator.remove]}), and doesn't implement BlockingQueue. That means 
{{Semaphore}} or something is required for exclusive control after all.

 a single callQueue related improvements
 ---

 Key: HBASE-14331
 URL: https://issues.apache.org/jira/browse/HBASE-14331
 Project: HBase
  Issue Type: Improvement
  Components: IPC/RPC, Performance
Reporter: Hiroshi Ikeda
Priority: Minor
 Attachments: CallQueuePerformanceTestApp.java


 {{LinkedBlockingQueue}} well separates locks between the {{take}} method and 
 the {{put}} method, but not between takers, and not between putters. These 
 methods are implemented to take locks at the almost beginning of their logic. 
 HBASE-11355 introduces multiple call-queues to reduce such possible 
 congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
 There are the other shortcomings of using {{BlockingQueue}}. When using 
 multiple queues, since {{BlockingQueue}} blocks threads it is required to 
 prepare enough threads for each queue. It is possible that there is a queue 
 starving for threads while there is another queue where threads are idle. 
 Even if you can tune parameters to avoid such situations, the tuning is not 
 so trivial.
 I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table


 [ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick.han updated HBASE-14332:
-
Attachment: HBASE-14332_v3.patch

this patch fixed typo and remove commented out code.

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch, 
 HBASE-14332_v3.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13161) ITBLL needs love; verify spins w/o end at 10B rows

2015-08-30 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-13161.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 1.2.0

Resolving. Did an ITBLL 10B run w/ monkeys. Passed. See below. The change to 
use securerandom seems to have gotten us over the observed hang.

Running to 10B trying to save all hfiles and WALs made in case I find an issue 
so I can dig in if verify fails is beyond the disk capacity I have on my small 
cluster; I had to remove a load of archived material to run verify.

Verify took 18 hours.

Will come back to this to go larger when I have more capacity. Need to also 
make verify run faster.

{code}
15/08/29 14:58:54 INFO mapreduce.Job: Job job_1440806964191_0001 completed 
successfully
15/08/29 14:58:54 INFO mapreduce.Job: Counters: 51
File System Counters
FILE: Number of bytes read=3119022047178
FILE: Number of bytes written=3814321075643
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2484
HDFS: Number of bytes written=480
HDFS: Number of read operations=55
HDFS: Number of large read operations=0
HDFS: Number of write operations=10
Job Counters
Launched map tasks=20
Launched reduce tasks=5
Data-local map tasks=19
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=405431651
Total time spent by all reduces in occupied slots (ms)=463831492
Total time spent by all map tasks (ms)=405431651
Total time spent by all reduce tasks (ms)=231915746
Total vcore-seconds taken by all map tasks=405431651
Total vcore-seconds taken by all reduce tasks=231915746
Total megabyte-seconds taken by all map tasks=1621726604000
Total megabyte-seconds taken by all reduce tasks=1855325968000
Map-Reduce Framework
Map input records=100
Map output records=200
Map output bytes=6500
Map output materialized bytes=69000600
Input split bytes=2484
Combine input records=0
Combine output records=0
Reduce input groups=100
Reduce shuffle bytes=69000600
Reduce input records=200
Reduce output records=0
Spilled Records=110403900880
Shuffled Maps =100
Failed Shuffles=0
Merged Map outputs=100
GC time elapsed (ms)=2193043
CPU time spent (ms)=390646120
Physical memory (bytes) snapshot=23430848512
Virtual memory (bytes) snapshot=183185506304
Total committed heap usage (bytes)=21181235200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify$Counts
REFERENCED=100
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=480
{code}

 ITBLL needs love; verify spins w/o end at 10B rows
 --

 Key: HBASE-13161
 URL: https://issues.apache.org/jira/browse/HBASE-13161
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Reporter: stack
Assignee: stack
 Fix For: 1.2.0


 Making a note of my recent experience with ITBLL; here is stuff that needs 
 fixing.
 + I was able to run generate for about 24 hours with monkey going and loaded 
 10B rows. When I ran the verify (a few times), it got stuck at about 90odd 
 percent and never made progress beyond that (this was about 20 hours in 
 IIRC). I spent no time trying to figure why (ran out of cluster and time).
 + Verify at least takes too long. Need to make it run faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table


[ 
https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14722711#comment-14722711
 ] 

Nick.han commented on HBASE-14332:
--

Thank you very much Ted.Your suggestion is very helpful  to me.

 Show the table state when we encounter exception while disabling / enabling 
 table
 -

 Key: HBASE-14332
 URL: https://issues.apache.org/jira/browse/HBASE-14332
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Nick.han
Assignee: Nick.han
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-14332.patch, HBASE-14332_v2.patch


 This patch is a advice for good user experience
 reason:
 When we disable a table and the table is not enabled,we receive a 
 exception,but the exception is too brief,some time we  want to know what 
 state is the table in,so that we can know why the table can't be disable.For 
 example,I once encountered a problem the table is neither disable nor enable 
 when my region server crash down,if we give the table state,I will find the 
 problem more quickly .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table