[jira] [Commented] (HBASE-17816) HRegion#mutateRowWithLocks should update writeRequestCount metric

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959986#comment-15959986
 ] 

Jerry He commented on HBASE-17816:
--

Pushed to master.
[~qgxiaozhan]  If you provide a branch-1 patch, I'll commit it.

> HRegion#mutateRowWithLocks should update writeRequestCount metric
> -
>
> Key: HBASE-17816
> URL: https://issues.apache.org/jira/browse/HBASE-17816
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Reporter: Ashu Pachauri
>Assignee: Weizhan Zeng
> Attachments: HBASE-17816.master.001.patch, 
> HBASE-17816.master.002.patch
>
>
> Currently, all the calls that use HRegion#mutateRowWithLocks miss 
> writeRequestCount metric. The mutateRowWithLocks base method should update 
> the metric.
> Examples are checkAndMutate calls through RSRpcServices#multi, 
> Region#mutateRow api , MultiRowMutationProcessor coprocessor endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959980#comment-15959980
 ] 

Jerry He commented on HBASE-17869:
--

Filed HBASE-17890.

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17890) FuzzyRow tests fail if unaligned support is false

[Jerry He (JIRA)]

Jerry He created HBASE-17890:


 Summary: FuzzyRow tests fail if unaligned support is false
 Key: HBASE-17890
 URL: https://issues.apache.org/jira/browse/HBASE-17890
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.5, 2.0.0
Reporter: Jerry He


When unaligned support is false, FuzzyRow tests fail:
{noformat}
Failed tests:
  TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186 
expected:<10> but was:<0>
  TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
  TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but 
was:
  TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343 
expected:<6250> but was:<0>
  TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445 
expected:<5> but was:<0>
  TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
{noformat}
This can be reproduced in the case described in HBASE-17869. Or on a platform 
really without unaligned support.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17869:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959964#comment-15959964
 ] 

Jerry He commented on HBASE-17869:
--

Pushed to master and branch-1. (Fixed Ted's nit.)
Tested on ppc64le: 
Without the patch:
{code}
Failed tests:
  TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186 
expected:<10> but was:<0>
  TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
  TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but 
was:
  TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343 
expected:<6250> but was:<0>
  TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445 
expected:<5> but was:<0>
  TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
{code}
With the patch,  the full 'mvn clean test' is clean on master :-)

Thanks for the review.

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958287#comment-15958287
 ] 

Jerry He commented on HBASE-17869:
--

The spark fix does the same exact string match. But it covers more platforms 
with an extra pattern matching and for fall-back checking.
https://github.com/apache/spark/pull/17472/files
It is probably ok that we don't have the extra checking.

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17878) java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter when starting HBase with hbase.rootdir on S3

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957863#comment-15957863
 ] 

Jerry He commented on HBASE-17878:
--

FYI [~zyork]

> java.lang.NoSuchMethodError: 
> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter
>  when starting HBase with hbase.rootdir on S3
> -
>
> Key: HBASE-17878
> URL: https://issues.apache.org/jira/browse/HBASE-17878
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> When setting up HBASE-17437 (Support specifying a WAL directory outside of 
> the root directory), we specify
> (1) hbase.rootdir on s3a
> (2) hbase.wal.dir on HDFS
> When starting HBase, the following exception is thrown:
> {code}
> Caused by: java.lang.NoSuchMethodError: 
> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
> at 
> com.amazonaws.auth.internal.AWS4SignerUtils.(AWS4SignerUtils.java:26)
> at 
> com.amazonaws.auth.internal.AWS4SignerRequestParams.(AWS4SignerRequestParams.java:85)
> at com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:184)
> at 
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:709)
> at 
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at 
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at 
> com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
> at 
> com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
> at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:232)
> at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:1007)
> at 
> org.apache.hadoop.hbase.util.FSUtils.isValidWALRootDir(FSUtils.java:1050)
> at 
> org.apache.hadoop.hbase.util.FSUtils.getWALRootDir(FSUtils.java:1032)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:627)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:570)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:393)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2456)
> ... 5 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956312#comment-15956312
 ] 

Jerry He commented on HBASE-17869:
--

I found out SPARK-1 does something similar.

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956310#comment-15956310
 ] 

Jerry He commented on HBASE-17869:
--

Thanks for chiming in. Do you agree I can commit this one and open another 
issue to look into the false path with FuzzyRowFilter?

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956062#comment-15956062
 ] 

Jerry He commented on HBASE-17869:
--

On a x86_64 box, if I hard-code UnsafeAvailChecker.unaligned() to false, the 
same tests fail.


> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955986#comment-15955986
 ] 

Jerry He commented on HBASE-17869:
--

Yeah, I had the same question. Two code paths (true or false) should yield the 
same results.
Since the 'true' route is more commonly used, can we assume it has the correct 
results?
Then there could be a bug somewhere in the 'false' route. In the FuzzyRowFilter?
{code}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hbase.filter.TestFuzzyRowFilter.testSatisfiesForward(TestFuzzyRowFilter.java:80)
{code}
{code}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hbase.filter.TestFuzzyRowFilter.testSatisfiesReverse(TestFuzzyRowFilter.java:120)
{code}


> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955436#comment-15955436
 ] 

Jerry He commented on HBASE-17861:
--

+1

> Regionserver down when checking the permission of staging dir if 
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>  Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, 
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch, 
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory 
> outside of the root directory.
> The region server are  showdown when I add following config into 
> hbase-site.xml 
> hbase.rootdir =  s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes = 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw 
> java.lang.IllegalStateException: Directory already exists but permissions 
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a 
> folder in s3, it can not set above permission, because in s3, all files are 
> listed as having full read/write permissions and all directories appear to 
> have full rwx permissions. See Object stores have differerent authorization 
> models in 
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954577#comment-15954577
 ] 

Jerry He commented on HBASE-17857:
--

Is it the case that from now on, we will say we will mark unstable new feature 
API as IA private intentionally?

> Remove IS annotations from IA.Public classes
> 
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17857.patch, HBASE-17857-v1.patch, 
> HBASE-17857-v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17869:
-
Fix Version/s: 1.4.0
   2.0.0
   Status: Patch Available  (was: Open)

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17869:
-
Attachment: HBASE-17869.patch

> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

Jerry He created HBASE-17869:


 Summary: UnsafeAvailChecker wrongly returns false on ppc
 Key: HBASE-17869
 URL: https://issues.apache.org/jira/browse/HBASE-17869
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.4
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor


On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
bug.
https://bugs.openjdk.java.net/browse/JDK-8165231
This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952042#comment-15952042
 ] 

Jerry He commented on HBASE-17861:
--

We can just these three: s3, wasb and swift, as [~zyork] suggested, and ignore 
case.

> Regionserver down when checking the permission of staging dir if 
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>  Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory 
> outside of the root directory.
> The region server are  showdown when I add following config into 
> hbase-site.xml 
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes = 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw 
> java.lang.IllegalStateException: Directory already exists but permissions 
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a 
> folder in s3, it can not set above permission, because in s3, all files are 
> listed as having full read/write permissions and all directories appear to 
> have full rwx permissions. See Object stores have differerent authorization 
> models in 
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952025#comment-15952025
 ] 

Jerry He commented on HBASE-16179:
--

bq. I'd be up for us changing gears a bit here. We could focus here just on 
what has to be done for correctness and do things that would be nice in 
follow-ons
Agree.

bq.  I do agree that I'd like Spark 2 support for HBase 2.0.0.
Agree too.

bq. Just make sure that however those classpaths are formed we don't include 
the various sparkXscala specific jars.
Confused here.  Any reason why we don't want one to be in the classpath of the 
server? I think there is a custom filter in the jar that needs to be on the 
server.


> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, 
> 16179.v25.txt, 16179.v26.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 
> 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951959#comment-15951959
 ] 

Jerry He commented on HBASE-17857:
--

There are a few scala source files in hbase-spark you may want to include in 
your script.

> Remove IS annotations from IA.Public classes
> 
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17857.patch, HBASE-17857-v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951728#comment-15951728
 ] 

Jerry He commented on HBASE-17861:
--

bq. It seems like this should be the exception instead of the rule.

Makes sense.

> Regionserver down when checking the permission of staging dir if 
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>  Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory 
> outside of the root directory.
> The region server are  showdown when I add following config into 
> hbase-site.xml 
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes = 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw 
> java.lang.IllegalStateException: Directory already exists but permissions 
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a 
> folder in s3, it can not set above permission, because in s3, all files are 
> listed as having full read/write permissions and all directories appear to 
> have full rwx permissions. See Object stores have differerent authorization 
> models in 
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17816) HRegion#mutateRowWithLocks should update writeRequestCount metric

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949538#comment-15949538
 ] 

Jerry He commented on HBASE-17816:
--

+1

> HRegion#mutateRowWithLocks should update writeRequestCount metric
> -
>
> Key: HBASE-17816
> URL: https://issues.apache.org/jira/browse/HBASE-17816
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Reporter: Ashu Pachauri
>Assignee: Weizhan Zeng
> Attachments: HBASE-17816.master.001.patch, 
> HBASE-17816.master.002.patch
>
>
> Currently, all the calls that use HRegion#mutateRowWithLocks miss 
> writeRequestCount metric. The mutateRowWithLocks base method should update 
> the metric.
> Examples are checkAndMutate calls through RSRpcServices#multi, 
> Region#mutateRow api , MultiRowMutationProcessor coprocessor endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17827) Client tools relying on AuthUtil.getAuthChore() break credential cache login

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944273#comment-15944273
 ] 

Jerry He commented on HBASE-17827:
--

Sounds good.

> Client tools relying on AuthUtil.getAuthChore() break credential cache login
> 
>
> Key: HBASE-17827
> URL: https://issues.apache.org/jira/browse/HBASE-17827
> Project: HBase
>  Issue Type: Bug
>  Components: canary, security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
>
> Client tools, such as Canary, which make use of keytab based logins with 
> AuthUtil.getAuthChore() do not allow any way to continue without a 
> keytab-based login when security is enabled.  Currently, when security is 
> enabled and the configuration lacks {{hbase.client.keytab.file}}, these tools 
> would fail with:
> {noformat}
> ERROR hbase.AuthUtil: Error while trying to perform the initial login: 
> Running in secure mode, but config doesn't have a keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at 
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
> at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
> at org.apache.hadoop.hbase.security.User.login(User.java:258)
> at 
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
> at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
> at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
> Exception in thread "main" java.io.IOException: Running in secure mode, but 
> config doesn't have a keytab
> at 
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
> at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
> at org.apache.hadoop.hbase.security.User.login(User.java:258)
> at 
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
> at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
> at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
> {noformat}
> These tools should still work with the default credential-cache login, at 
> least when a client keytab is not configured.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17827) Client tools relying on AuthUtil.getAuthChore() break credential cache login

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939473#comment-15939473
 ] 

Jerry He commented on HBASE-17827:
--

I think either of your approaches looks fine.
Are you still going to use the same chore mechanism to re-login, from cache, 
even the cache has limited lifetime?

> Client tools relying on AuthUtil.getAuthChore() break credential cache login
> 
>
> Key: HBASE-17827
> URL: https://issues.apache.org/jira/browse/HBASE-17827
> Project: HBase
>  Issue Type: Bug
>  Components: canary, security
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
>
> Client tools, such as Canary, which make use of keytab based logins with 
> AuthUtil.getAuthChore() do not allow any way to continue without a 
> keytab-based login when security is enabled.  Currently, when security is 
> enabled and the configuration lacks {{hbase.client.keytab.file}}, these tools 
> would fail with:
> {noformat}
> ERROR hbase.AuthUtil: Error while trying to perform the initial login: 
> Running in secure mode, but config doesn't have a keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at 
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
> at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
> at org.apache.hadoop.hbase.security.User.login(User.java:258)
> at 
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
> at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
> at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
> Exception in thread "main" java.io.IOException: Running in secure mode, but 
> config doesn't have a keytab
> at 
> org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
> at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:420)
> at org.apache.hadoop.hbase.security.User.login(User.java:258)
> at 
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:197)
> at org.apache.hadoop.hbase.AuthUtil.getAuthChore(AuthUtil.java:98)
> at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:589)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1327)
> {noformat}
> These tools should still work with the default credential-cache login, at 
> least when a client keytab is not configured.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932839#comment-15932839
 ] 

Jerry He commented on HBASE-16179:
--

Hi, Ted and Sean

I think it will as well be a good topic to bring it up to the dev and user 
lists for a discussion and awareness.

> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, 
> 16179.v25.txt, 16179.v26.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 
> 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17803) PE always re-creates table when we specify the split policy

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931051#comment-15931051
 ] 

Jerry He commented on HBASE-17803:
--

+1

> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
>   || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName() 
> != opts.splitPolicy)
>   || (!isReadCmd && desc != null && desc.getRegionReplication() != 
> opts.replicas)) {
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930336#comment-15930336
 ] 

Jerry He commented on HBASE-16247:
--

Just want to clarify a little more.  Hope I am right.  As soon as the hbase 
spark data source implementation sees the Avro schema as a column, it converts 
the Avro types to Spark SQL types. ENUM is mapped to StringType. After that, 
ENUM will just disappear throughout the catalyst data flow, including 
serialization or deserialization in the hbase spark data source.

> SparkSQL Avro serialization doesn't handle enums correctly
> --
>
> Key: HBASE-16247
> URL: https://issues.apache.org/jira/browse/HBASE-16247
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> Avro's generic api expects GenericEnumSymbol as the runtime type for 
> instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are 
> lax in some cases for handling this, but the 1.8 libraries are strict. We 
> should proactively fix our serialization.
> (the lax serialization in 1.7 fails for some nested use in unions, see 
> AVRO-997 for details)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927158#comment-15927158
 ] 

Jerry He commented on HBASE-16247:
--

Looking at the hbase spark code.  It does not seem to be a problem because 
hbase spark sql does not construct a Record from Avro type ENUM directly if I 
look at it right.

> SparkSQL Avro serialization doesn't handle enums correctly
> --
>
> Key: HBASE-16247
> URL: https://issues.apache.org/jira/browse/HBASE-16247
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> Avro's generic api expects GenericEnumSymbol as the runtime type for 
> instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are 
> lax in some cases for handling this, but the 1.8 libraries are strict. We 
> should proactively fix our serialization.
> (the lax serialization in 1.7 fails for some nested use in unions, see 
> AVRO-997 for details)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927153#comment-15927153
 ] 

Jerry He commented on HBASE-16247:
--

Hi, [~busbey]  Can you provide more details on the problem with hbase spark sql 
handling of the Avro enum type?

> SparkSQL Avro serialization doesn't handle enums correctly
> --
>
> Key: HBASE-16247
> URL: https://issues.apache.org/jira/browse/HBASE-16247
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> Avro's generic api expects GenericEnumSymbol as the runtime type for 
> instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are 
> lax in some cases for handling this, but the 1.8 libraries are strict. We 
> should proactively fix our serialization.
> (the lax serialization in 1.7 fails for some nested use in unions, see 
> AVRO-997 for details)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923589#comment-15923589
 ] 

Jerry He commented on HBASE-14161:
--

Ok. The tests in hbase-spark are run as regular unit tests now after 
HBASE-17574.
We may need to write new integration tests for the spark module. But until that 
is available, there is real IT yet.

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922888#comment-15922888
 ] 

Jerry He commented on HBASE-16179:
--

The SparkSQLPushDownFilter makes the hbase-spark jar necessary in hbase lib.

> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, 
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922885#comment-15922885
 ] 

Jerry He commented on HBASE-16179:
--

Adding to the argument what spark/scala version combinations we want to support 
out of the box, it does not seem practical that we have four binary 
distribution tarballs.


> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, 
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922780#comment-15922780
 ] 

Jerry He commented on HBASE-16179:
--

Hi, Ted

I also meant the binary tarball assembly.
I did with the v23 patch:
mvn package install assembly:single -DskipTests

I could not see any hbase-spark jars or the compat jars in the lib dir.
We need one and only version of the jars in the binary tarball.


> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v4.txt, 
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922746#comment-15922746
 ] 

Jerry He commented on HBASE-14161:
--

Let me take a look. Maybe I misunderstood the JIRA.

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-15597) Clean up configuration keys used in hbase-spark module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-15597:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the contribution and review.  Pushed to master.

> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, 
> HBASE-15597-V3.patch, HBase-15597-V4.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will 
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since 
> our configurations keys for the spark integration all relate to that system, 
> the prefix for all configuration keys (excluding those cases where we need to 
> do something special due to restrictions in how properties are handled by 
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we 
> should review all of our configuration keys to make sure they either conform 
> to the "hbase.spark" prefix or they have a comment documenting why they need 
> to be otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14161.
--
Resolution: Duplicate

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reopened HBASE-14161:
--

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14161.
--
Resolution: Fixed

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14167.
--
   Resolution: Duplicate
Fix Version/s: 2.0.0

> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
>  Issue Type: Improvement
>  Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14167.11.patch, HBASE-14167-master-v2.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark 
> module's integration tests do not respect the flag and run anyway. Fix. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906896#comment-15906896
 ] 

Jerry He commented on HBASE-15597:
--

Ping [~busbey] as well.
If there is no more comment, I will commit it tomorrow.
Thanks.

> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, 
> HBASE-15597-V3.patch, HBase-15597-V4.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will 
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since 
> our configurations keys for the spark integration all relate to that system, 
> the prefix for all configuration keys (excluding those cases where we need to 
> do something special due to restrictions in how properties are handled by 
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we 
> should review all of our configuration keys to make sure they either conform 
> to the "hbase.spark" prefix or they have a comment documenting why they need 
> to be otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906895#comment-15906895
 ] 

Jerry He commented on HBASE-16179:
--

Hi, Ted

Sorry for the delay in review.  I think it overall looks good!  Most of the 
things are in place.

A few comments.

[~busbey] mentioned about the assembly tarball in his previous comment.  If my 
understanding is correct, each assembly tarball can only include one version 
the hbase spark jar.
I didn't see how we accomplish it.

Also an overall comment on the supported combinations. We have four and we are 
building and possibly deploying to maven repo all of these four.  My thinking 
is that it seems to be an overkill, We are a HBase project, not a Spark or 
Scala project.  What about we have two compat module: one for spark 1.6 and one 
for spark 2?  Spark 2 defaults with Scala 2.11 out of box.  Spark 1.6 defaults 
with Scala 2.10 out of box.  

We claim support for the other two combinations.  But if needed, user needs to 
compile from source with -Dscala.version=xxx to get the one that is not 
available out of the box.
That will simply our build and distribution.
My 2 cents.

> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, 
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v4.txt, 
> 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905863#comment-15905863
 ] 

Jerry He commented on HBASE-15597:
--

+1 on v4.

> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, 
> HBASE-15597-V3.patch, HBase-15597-V4.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will 
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since 
> our configurations keys for the spark integration all relate to that system, 
> the prefix for all configuration keys (excluding those cases where we need to 
> do something special due to restrictions in how properties are handled by 
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we 
> should review all of our configuration keys to make sure they either conform 
> to the "hbase.spark" prefix or they have a comment documenting why they need 
> to be otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905501#comment-15905501
 ] 

Jerry He commented on HBASE-16010:
--

I was trying to give myself more time to think, but forgot to get back to it.  
Sorry about it.
The current patch provides API via master to put region servers in 'drain' mode 
indicating no regions should be assigned here.
The argument is whether this JIRA should include additional work to actually 
move the regions out from the affected server, again carried out on master, 
thus providing a true 'decommission' feature, which is not available anywhere 
yet.  A rb script currently does part of the work.
I like the idea. If the contributor will extend the work, I will be glad to 
help.
There was concern doing the bulk move of the regions on the master at the 
moment given the ongoing work in the AM proc.

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch, 
> HBASE-16010-v3.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904286#comment-15904286
 ] 

Jerry He commented on HBASE-15597:
--

v3 LGTM

{code}
val QUERY_CACHESIZE = "hbase.spark.query.cachesize"
val QUERY_BATCHSIZE = "hbase.spark.query.batchsize"
{code}
{code}
val TIMERANGE_START = "hbase.spark.query.timerange.start"
val TIMERANGE_END =  "hbase.spark.query.timerange.end"
val MAX_VERSIONS = "hbase.spark.query.maxVersions"
val MAX_VERSIONS = "hbase.spark.query.maxVersions"
{code}

It is good we use 'query' instead of 'scan' in the context of Spark query.
'cachesize' is named 'cachedrows' in TableInputFormat.  Let's follow it?
{code}
/** The number of rows for caching that will be passed to scanners. */
  public static final String SCAN_CACHEDROWS = 
"hbase.mapreduce.scan.cachedrows";
{code}
Please add some comments on these names if you can.

> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, 
> HBASE-15597-V3.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will 
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since 
> our configurations keys for the spark integration all relate to that system, 
> the prefix for all configuration keys (excluding those cases where we need to 
> do something special due to restrictions in how properties are handled by 
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we 
> should review all of our configuration keys to make sure they either conform 
> to the "hbase.spark" prefix or they have a comment documenting why they need 
> to be otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904021#comment-15904021
 ] 

Jerry He commented on HBASE-16598:
--

Hi, [~syuanjiang] 

It is probably for the incompatible change mentioned in the Release Notes.

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598.patch, HBASE-16598-v2.patch, 
> HBASE-16598-v3.patch, HBASE-16598-v4.patch, HBASE-16598-v5.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17701) Add HadoopAuthFilterInitializer to use hadoop-auth AuthenticationFilter for hbase web ui

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902349#comment-15902349
 ] 

Jerry He commented on HBASE-17701:
--

Hi, [~panyuxuan]

I see what you try to accomplish. But it is not necessary to pull in 
HadoopAuthFilterInitializer and to use the hadoop.http.authentication.* 
properties.

HBASE-5291's implementation is consistent with the hadoop htttp way since it 
uses the hadoop AuthenticationFilter.  The thing is that the parameter to 
AuthenticationFilter is hard set to 'kerberos' if the hbase ui security in on.  
hadoop AuthenticationFilter is designed to be more flexible in the sense that 
it can accept other AuthenticationHandler implementation, i.e. custom 
authentication class name. 
[Here|https://hadoop.apache.org/docs/r2.7.2/hadoop-auth/Configuration.html].
You should be able to provide a patch that improves and make the current 
implementation flexible instead of entirely pulling in 
HadoopAuthFilterInitializer. 

> Add HadoopAuthFilterInitializer to use hadoop-auth AuthenticationFilter for 
> hbase web ui
> 
>
> Key: HBASE-17701
> URL: https://issues.apache.org/jira/browse/HBASE-17701
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Affects Versions: 1.2.4
>Reporter: Pan Yuxuan
> Attachments: HBASE-17701.v1.patch
>
>
> The HBase web UI is none secure by default, there is only one 
> StaticUserWebFilter for a fake user.
> For Hadoop, we already have AuthenticationFilter for web authentication based 
> on token or kerberos. So I think hbase can reuse the hadoop-auth 
> AuthenticationFilter by adding a HadoopAuthFilterInitializer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HBASE-15335) Add composite key support in row key

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-15335:


Assignee: Jerry He

> Add composite key support in row key
> 
>
> Key: HBASE-15335
> URL: https://issues.apache.org/jira/browse/HBASE-15335
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Zhan Zhang
>Assignee: Jerry He
> Attachments: HBASE-15335-1.patch, HBASE-15335-2.patch, 
> HBASE-15335-3.patch, HBASE-15335-4.patch, HBASE-15335-5.patch, 
> HBASE-15335-6.patch, HBASE-15335-7.patch, HBASE-15335-8.patch, 
> HBASE-15335-9.patch
>
>
> Add composite key filter support in the connector.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15335) Add composite key support in row key

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902187#comment-15902187
 ] 

Jerry He commented on HBASE-15335:
--

Thanks for the work, [~zhanzhang].

> Add composite key support in row key
> 
>
> Key: HBASE-15335
> URL: https://issues.apache.org/jira/browse/HBASE-15335
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Zhan Zhang
> Attachments: HBASE-15335-1.patch, HBASE-15335-2.patch, 
> HBASE-15335-3.patch, HBASE-15335-4.patch, HBASE-15335-5.patch, 
> HBASE-15335-6.patch, HBASE-15335-7.patch, HBASE-15335-8.patch, 
> HBASE-15335-9.patch
>
>
> Add composite key filter support in the connector.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17736) Some options can't be configured by the shell

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896511#comment-15896511
 ] 

Jerry He commented on HBASE-17736:
--

See HBASE-17729.

> Some options can't be configured by the shell
> -
>
> Key: HBASE-17736
> URL: https://issues.apache.org/jira/browse/HBASE-17736
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17736.v0.patch
>
>
> The lost options for table are shown below.
> # setFlushPolicyClassName
> # setPriority
> # setRegionMemstoreReplication
> # setRegionSplitPolicyClassName
> The lost options for column are shown below.
> # setCacheBloomsOnWrite
> # setCacheDataInL1
> # setCacheIndexesOnWrite
> # setCompactionCompressionType
> # setEvictBlocksOnClose



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14375) Define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14375:
-
Summary: Define public API for spark integration module  (was: define 
public API for spark integration module)

> Define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch, HBASE-14375-v2.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17729) Missing shortcuts for some useful HCD options

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896084#comment-15896084
 ] 

Jerry He commented on HBASE-17729:
--

+1

> Missing shortcuts for some useful HCD options
> -
>
> Key: HBASE-17729
> URL: https://issues.apache.org/jira/browse/HBASE-17729
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 1.3.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: HBASE-17729.patch
>
>
> Missing:
> - CACHE_INDEX_ON_WRITE 
> - CACHE_BLOOMS_ON_WRITE
> - EVICT_BLOCKS_ON_CLOSE
> - CACHE_DATA_IN_L1
> It's possible to set them with syntax like 
> {noformat}
> { CONFIGURATION = {  =  } }
> {noformat}
> but let's save some keystrokes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896022#comment-15896022
 ] 

Jerry He commented on HBASE-17718:
--

LGTM on patch 002.  Minor comment on RB.

> Difference between RS's servername and its ephemeral node cause SSH stop 
> working
> 
>
> Key: HBASE-17718
> URL: https://issues.apache.org/jira/browse/HBASE-17718
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.4, 1.1.8
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17718.master.001.patch, 
> HBASE-17718.master.002.patch
>
>
> After HBASE-9593, RS put up an ephemeral node in ZK before reporting for 
> duty. But if the hosts config (/etc/hosts) is different between master and 
> RS, RS's serverName can be different from the one stored the ephemeral zk 
> node. The email metioned in HBASE-13753 
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox/%3CCANZDn9ueFEEuZMx=pZdmtLsdGLyZz=rrm1N6EQvLswYc1z-H=g...@mail.gmail.com%3E)
>  is exactly what happened in our production env. 
> But what the email didn't point out is that the difference between serverName 
> in RS and zk node can cause SSH stop to work. as we can see from the code in 
> {{RegionServerTracker}}
> {code}
>   @Override
>   public void nodeDeleted(String path) {
> if (path.startsWith(watcher.rsZNode)) {
>   String serverName = ZKUtil.getNodeName(path);
>   LOG.info("RegionServer ephemeral node deleted, processing expiration [" 
> +
> serverName + "]");
>   ServerName sn = ServerName.parseServerName(serverName);
>   if (!serverManager.isServerOnline(sn)) {
> LOG.warn(serverName.toString() + " is not online or isn't known to 
> the master."+
>  "The latter could be caused by a DNS misconfiguration.");
> return;
>   }
>   remove(sn);
>   this.serverManager.expireServer(sn);
> }
>   }
> {code}
> The server will not be processed by SSH/ServerCrashProcedure. The regions on 
> this server will not been assigned again until master restart or failover.
> I know HBASE-9593 was to fix the issue if RS report to duty and crashed 
> before it can put up a zk node. It is a very rare case(And controllable， just 
> fix the bug making rs to crash). But The issue I metioned can happened more 
> often(and uncontrollable, can't be fixed in HBase, due to DNS, hosts config, 
> etc.) and have more severe consequence.
> So here I offer some solutions to discuss:
> 1. Revert HBASE-9593 from all branches, Andrew Purtell has reverted it in 
> branch-0.98
> 2. Abort RS if master return a different name, otherwise SSH can't work 
> properly
> 3. Master accepts whatever servername reported by RS and don't change it.
> 4.correct the zk node if master return another name( idea from Ted Yu)
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895899#comment-15895899
 ] 

Jerry He commented on HBASE-14375:
--

Sounds good. Thanks for the review. 
I kept DefaultSource as IA.Private, but added into the class comment:

+ * This class needs to stay in the current package 
'org.apache.hadoop.hbase.spark'
+ * for Spark to match the hbase data source name.

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch, HBASE-14375-v2.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14375:
-
Attachment: HBASE-14375-v2.patch

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch, HBASE-14375-v2.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14375:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17722) Metrics subsystem stop/start messages add a lot of useless bulk to operational logging

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893688#comment-15893688
 ] 

Jerry He commented on HBASE-17722:
--

I too noticed these messages showing up in recent version of 1.2.x

{noformat}
2017-02-27 17:21:00,974 INFO  [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink 
timeline started
2017-02-27 17:21:00,976 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: 
Scheduled snapshot period at 10 second(s).
2017-02-27 17:21:00,976 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase 
metrics system started
{noformat}
{noformat}
2017-02-27 17:36:00,449 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: 
Stopping HBase metrics system...
2017-02-27 17:36:00,450 INFO  [timeline] impl.MetricsSinkAdapter: timeline 
thread interrupted.
2017-02-27 17:36:00,455 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase 
metrics system stopped.
{noformat}

Do you want to add impl.MetricsSinkAdapter?  Do you want to put it as 'WARN'?
Otherwise  +1

> Metrics subsystem stop/start messages add a lot of useless bulk to 
> operational logging
> --
>
> Key: HBASE-17722
> URL: https://issues.apache.org/jira/browse/HBASE-17722
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.3.0, 1.2.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17722.patch
>
>
> Metrics subsystem stop/start messages add a lot of useless bulk to 
> operational logging. Say you are collecting logs from a fleet of thousands of 
> servers and want to have them around for ~month or longer. It adds up. 
> I think these should at least be at DEBUG level and ideally at TRACE. They 
> don't offer much utility. Unfortunately they are Hadoop classes so we can 
> tweak log4j.properties defaults instead. We do this in test resources but not 
> in what we ship in conf/ . 
> {noformat}
>  INFO  [] impl.MetricsSystemImpl: HBase metrics system started
>  INFO  [] impl.MetricsSystemImpl: Stopping HBase metrics 
> system...
>  INFO  [] impl.MetricsSystemImpl: HBase metrics system stopped.
>  INFO  [] impl.MetricsConfig: loaded properties from 
> hadoop-metrics2-hbase.properties
>  INFO  [] impl.MetricsSystemImpl: Scheduled snapshot period at 
> 10 second(s).
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893544#comment-15893544
 ] 

Jerry He commented on HBASE-14375:
--

Regarding the API docs, is it our goal that we convert the scala-doc to 
java-doc and show them together with the rest of HBase Java API docs?
Hopefully the annotations can be effectively filtered as they are now.  
Otherwise we have to find other ways to do it. For example, going into the 
scala classes to update the class/function modifiers?  Hopefully that can be 
separated into another task.

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893539#comment-15893539
 ] 

Jerry He commented on HBASE-14375:
--

I can:
-- Use the DataSourceRegister trait.  Define the format name in the public 
HBaseSparkConf.  Still mark the data source implementation class private.
Or
-- Mark it LimitedPrivate with HBaseInterfaceAudience.SPARK

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891531#comment-15891531
 ] 

Jerry He edited comment on HBASE-14375 at 3/3/17 1:25 AM:
--

Do you think we can just mark the DefaultSource private as it is?


was (Author: jinghe):
You don't we can make the DefaultSource private as it is?

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17720) Possible bug in FlushSnapshotSubprocedure

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892888#comment-15892888
 ] 

Jerry He commented on HBASE-17720:
--

It is handled in the upper level RegionServerSnapshotManager.

> Possible bug in FlushSnapshotSubprocedure
> -
>
> Key: HBASE-17720
> URL: https://issues.apache.org/jira/browse/HBASE-17720
> Project: HBase
>  Issue Type: Bug
>  Components: dataloss, snapshots
>Reporter: Ben Lau
>
> I noticed that FlushSnapshotSubProcedure differs from MemstoreFlusher in that 
> it does not appear to explicitly handle a DroppedSnapshotException.  In the 
> primary codepath when flushing memstores, (see 
> MemStoreFlusher.flushRegion()), there is a try/catch for 
> DroppedSnapshotException that will abort the regionserver to replay WALs to 
> avoid data loss.  I don't see this in FlushSnapshotSubProcedure.  Is this an 
> accidental omission or is there a reason this isn't present?  
> I'm not too familiar with procedure V1 or V2.  I assume it is the case that 
> if a participant dies that all other participants will terminate any 
> outstanding operations for the procedure?  If so and if this lack of 
> RS.abort() for DroppedSnapshotException is a bug, then it can't be fixed 
> naively otherwise I assume a failed flush on 1 region server could cause a 
> cascade of RS abortions on the cluster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891531#comment-15891531
 ] 

Jerry He commented on HBASE-14375:
--

You don't we can make the DefaultSource private as it is?

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891313#comment-15891313
 ] 

Jerry He commented on HBASE-14375:
--

Regarding the DefaultSource class, if I understand it correctly, for the data 
source format 'org.apache.hadoop.hbase.spark' we give to Spark SQL, Spark SQL 
will automatically look for 'DefaultSource' as the implementation class in the 
same package.  This can be changed with a call to Spark SQL 
'DataSourceRegister' -- an improvement we can do later.

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17701) Add HadoopAuthFilterInitializer to use hadoop-auth AuthenticationFilter for hbase web ui

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887484#comment-15887484
 ] 

Jerry He commented on HBASE-17701:
--

HBASE-5291

> Add HadoopAuthFilterInitializer to use hadoop-auth AuthenticationFilter for 
> hbase web ui
> 
>
> Key: HBASE-17701
> URL: https://issues.apache.org/jira/browse/HBASE-17701
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Affects Versions: 1.2.4
>Reporter: Pan Yuxuan
> Attachments: HBASE-17701.v1.patch
>
>
> The HBase web UI is none secure by default, there is only one 
> StaticUserWebFilter for a fake user.
> For Hadoop, we already have AuthenticationFilter for web authentication based 
> on token or kerberos. So I think hbase can reuse the hadoop-auth 
> AuthenticationFilter by adding a HadoopAuthFilterInitializer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14375:
-
Attachment: HBASE-14375-v1.patch

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-14375:
-
Status: Patch Available  (was: Open)

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14375-v1.patch
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884377#comment-15884377
 ] 

Jerry He commented on HBASE-14375:
--

Attached an initial patch trying to annotate the classes/functions in the spark 
module.

Went through each of the classes.  I tried to do the following:


-- HBaseContext  --> public, evolving.
-- HBase RDD functions (bulk get, bulk put, bulk load, stream, etc)  --> 
public, evolving.
-- NewHBaseRDD (wraps NewHadoopRDD)  --> public, evolving.
-- Some scala convenience/wrapper classes for hbase  --> public, evolving.
-- HBaseSparkConf, HBaseTableCatalog (for Spark SQL / DataFrame  datasource)  
--> public, evolving.
-- Data type encoder interface (for Spark SQL / DataFrame  datasource)  --> 
LimitedPrivate with HBaseInterfaceAudience.SPARK  --> Advanced user can have 
custom implementation.
-- Everything else, internal implementation of anything related to the 
datasource  --> private

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16990) Shell tool to dump table and namespace privileges

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884346#comment-15884346
 ] 

Jerry He commented on HBASE-16990:
--

The patch itself looks good.  The concern is that we try to consolidate into 
the shell because standalone rb scripts can become out-of-sync and stale 
quickly and no test coverage. Also there is substantial overlap in 
functionality with user_permission.
I am going to be neutral on this one. It's ok if people find it useful with 
this rb tool.

> Shell tool to dump table and namespace privileges
> -
>
> Key: HBASE-16990
> URL: https://issues.apache.org/jira/browse/HBASE-16990
> Project: HBase
>  Issue Type: New Feature
>  Components: tooling
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Minor
> Attachments: HBASE-16990.v1.patch, HBASE-16990.v2.patch, 
> hbase-site.xml
>
>
> Recently,  we are trying to migrate tables from Cluster-A to Cluster-B,  I 
> found that HBase lack some useful tools :
> 1.  dump table schema,  like mysqldump in mysql
> 2.  dump table privileges,  like pt-show-grants in mysql provided by Percona. 
> I think we can add a dump sub-command looks like (JUST simple demo) : 
> {code}
> $ ./bin/hbase dump  -t test_table  --with-privileges  > ~/test_table.hsh
> $ cat ~/test_table.hsh
> create 'test_table', {NAME=>'f1'} 
> grant 'test_user', 'RW', 'test_table'
> {code}
> Maybe I can contribute ...  :)
> How do you think ?  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16990) Shell tool to dump table and namespace privileges

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878957#comment-15878957
 ] 

Jerry He commented on HBASE-16990:
--

The shell command 'user_permission' does not work for you?  Why have a separate 
rb which we both need to maintain? 

> Shell tool to dump table and namespace privileges
> -
>
> Key: HBASE-16990
> URL: https://issues.apache.org/jira/browse/HBASE-16990
> Project: HBase
>  Issue Type: New Feature
>  Components: tooling
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Minor
> Attachments: HBASE-16990.v1.patch, HBASE-16990.v2.patch, 
> hbase-site.xml
>
>
> Recently,  we are trying to migrate tables from Cluster-A to Cluster-B,  I 
> found that HBase lack some useful tools :
> 1.  dump table schema,  like mysqldump in mysql
> 2.  dump table privileges,  like pt-show-grants in mysql provided by Percona. 
> I think we can add a dump sub-command looks like (JUST simple demo) : 
> {code}
> $ ./bin/hbase dump  -t test_table  --with-privileges  > ~/test_table.hsh
> $ cat ~/test_table.hsh
> create 'test_table', {NAME=>'f1'} 
> grant 'test_user', 'RW', 'test_table'
> {code}
> Maybe I can contribute ...  :)
> How do you think ?  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17648) HBase Table-level synchronization fails between two secured(kerberized) cluster

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866363#comment-15866363
 ] 

Jerry He commented on HBASE-17648:
--

+1

> HBase Table-level synchronization fails between two secured(kerberized) 
> cluster
> ---
>
> Key: HBASE-17648
> URL: https://issues.apache.org/jira/browse/HBASE-17648
> Project: HBase
>  Issue Type: Bug
>Reporter: Yi Liang
>Assignee: Yi Liang
> Attachments: HBASE-17648-V1.patch, HBASE-17648-V2.patch
>
>
> run org.apache.hadoop.hbase.mapreduce.SyncTable’ command.
> Follow the steps in HBASE-15557 between two secured cluster, get errors below:
> {code}
> WARN [hconnection-0x55b5e331-metaLookup-shared--pool6-t1] 
> org.apache.hadoop.hbase.ipc.RpcClientImpl: Exception encountered while 
> connecting to the server : javax.security.sasl.SaslException: GSS initiate 
> failed [Caused by GSSException: No valid credentials provided (Mechanism 
> level: Failed to find any Kerberos tgt)]
> 2017-02-08 14:33:57,465 FATAL 
> [hconnection-0x55b5e331-metaLookup-shared--pool6-t1] 
> org.apache.hadoop.hbase.ipc.RpcClientImpl: SASL authentication failed. The 
> most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
>   at 
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>   at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
>   at 
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
>   at 
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
>   at 
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>   at 
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
>   at 
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>   at 
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
>   at 
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>   at 
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>   at 
> 

[jira] [Commented] (HBASE-17574) Clean up how to run tests under hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856740#comment-15856740
 ] 

Jerry He commented on HBASE-17574:
--

Pushed to master.
Thanks for the reviews.
Thanks for the patch, [~easyliangjob].  A reminder to use 'git format-patch' to 
generate patch in the future.

> Clean up how to run tests under hbase-spark module 
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBase-17574-V1.patch, HBase-17574-V2.patch
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole 
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test, 
> i.e. we need to go to hbase-spark folder to use mvn verify to run the test 
> case.  I think these tests had better to be regard as unit test for the 
> following reasons:
> (1) All the scala test are very small, most of them can be finished within 
> 20s.
> (2) Integration test usually  put into hbase-it module, not in its own module.
> (3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop 
> QA will only run mvn test under root dir, however hbase-spark need mvn verify.
> (4) From its pom.xml below, you can see that, both 
> integration-test and test point to same 
> test. From MVN reference, 
> http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
>  we know that if a goal is bound to one or more build phases, that goal will 
> be called in all those phases. it means that mvn test and mvn 
> integration-test will do same thing, however true in 
> test phase just disable the mvn test command.  It is uncommon to have define 
> like that. 
> {code}
>   
> 
> test
> test
> 
> test
> 
> 
> true
> 
> 
> 
> integration-test
> integration-test
> 
> test
> 
> 
> Integration-Test
> 
> -Xmx1536m -XX:MaxPermSize=512m 
> -XX:ReservedCodeCacheSize=512m
> 
> false
> 
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855095#comment-15855095
 ] 

Jerry He commented on HBASE-16845:
--

Hi, [~yuzhih...@gmail.com]

This JIRA tries to do something similar to HBASE-17574. 
HBASE-17574 goes further to regard the hbase-spark tests as unit tests and 
eliminates the integration test phase run.

> Run tests under hbase-spark module in test phase
> 
>
> Key: HBASE-16845
> URL: https://issues.apache.org/jira/browse/HBASE-16845
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16845.v1.txt, 16845.v2.txt
>
>
> [~appy] reported over in the discussion thread titled 'Testing and CI' that 
> tests under hbase-spark module are not run by QA bot.
> This issue is to run the unit tests in test phase so that we detect build 
> breakage early.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HBASE-17574) Clean up how to run tests under hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853461#comment-15853461
 ] 

Jerry He edited comment on HBASE-17574 at 2/6/17 3:20 AM:
--

I applied the patch locally, ran the combination of commands listed in the 
first comment, with "mvn test -pl hbase-spark".
Everything looks good!  Both Java and scala tests.
I wonder why we had the tests in hbase-spark as integration tests in the first 
place.

+1




was (Author: jinghe):
I applied the patch locally, ran the combination of commands listed in the 
first comments, with "mvn test -pl hbase-spark".
Everything looks good!  Both Java and scala tests.
I wonder why we had the tests in hbase-spark as integration tests in the first 
place.

+1



> Clean up how to run tests under hbase-spark module 
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBase-17574-V1.patch
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole 
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test, 
> i.e. we need to go to hbase-spark folder to use mvn verify to run the test 
> case.  I think these tests had better to be regard as unit test for the 
> following reasons:
> (1) All the scala test are very small, most of them can be finished within 
> 20s.
> (2) Integration test usually  put into hbase-it module, not in its own module.
> (3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop 
> QA will only run mvn test under root dir, however hbase-spark need mvn verify.
> (4) From its pom.xml below, you can see that, both 
> integration-test and test point to same 
> test. From MVN reference, 
> http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
>  we know that if a goal is bound to one or more build phases, that goal will 
> be called in all those phases. it means that mvn test and mvn 
> integration-test will do same thing, however true in 
> test phase just disable the mvn test command.  It is uncommon to have define 
> like that. 
> {code}
>   
> 
> test
> test
> 
> test
> 
> 
> true
> 
> 
> 
> integration-test
> integration-test
> 
> test
> 
> 
> Integration-Test
> 
> -Xmx1536m -XX:MaxPermSize=512m 
> -XX:ReservedCodeCacheSize=512m
> 
> false
> 
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17574) Clean up how to run tests under hbase-spark module

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853461#comment-15853461
 ] 

Jerry He commented on HBASE-17574:
--

I applied the patch locally, ran the combination of commands listed in the 
first comments, with "mvn test -pl hbase-spark".
Everything looks good!  Both Java and scala tests.
I wonder why we had the tests in hbase-spark as integration tests in the first 
place.

+1



> Clean up how to run tests under hbase-spark module 
> ---
>
> Key: HBASE-17574
> URL: https://issues.apache.org/jira/browse/HBASE-17574
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBase-17574-V1.patch
>
>
> In master brunch, the test of hbase-spark module needs clean-up.
> I think we need to let hbase-spark follow the rules that exist in the whole 
> hbase project
> 1. In hbase-spark, all the scala test cases are regarded as integration test, 
> i.e. we need to go to hbase-spark folder to use mvn verify to run the test 
> case.  I think these tests had better to be regard as unit test for the 
> following reasons:
> (1) All the scala test are very small, most of them can be finished within 
> 20s.
> (2) Integration test usually  put into hbase-it module, not in its own module.
> (3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop 
> QA will only run mvn test under root dir, however hbase-spark need mvn verify.
> (4) From its pom.xml below, you can see that, both 
> integration-test and test point to same 
> test. From MVN reference, 
> http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
>  we know that if a goal is bound to one or more build phases, that goal will 
> be called in all those phases. it means that mvn test and mvn 
> integration-test will do same thing, however true in 
> test phase just disable the mvn test command.  It is uncommon to have define 
> like that. 
> {code}
>   
> 
> test
> test
> 
> test
> 
> 
> true
> 
> 
> 
> integration-test
> integration-test
> 
> test
> 
> 
> Integration-Test
> 
> -Xmx1536m -XX:MaxPermSize=512m 
> -XX:ReservedCodeCacheSize=512m
> 
> false
> 
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-17581) mvn clean test -PskipXXXTests does not work properly for some modules

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17581:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed.

> mvn clean test -PskipXXXTests does not work properly for some modules
> -
>
> Key: HBASE-17581
> URL: https://issues.apache.org/jira/browse/HBASE-17581
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBase-17581-V1.patch
>
>
> mvn clean test -PskipXXXTests will skip all tests in XXX  module,  it works 
> for hbase-server, when we use mvn clean test -PskipServerTests, 
> it will skip all small and medium tests in hbase-server module. 
> However for some module like hbase-common, this command only skip small 
> tests, the medium tests are still executed.
> Need to ensure all components work in the same way



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17581) mvn clean test -PskipXXXTests does not work properly for some modules

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850239#comment-15850239
 ] 

Jerry He commented on HBASE-17581:
--

LGTM

> mvn clean test -PskipXXXTests does not work properly for some modules
> -
>
> Key: HBASE-17581
> URL: https://issues.apache.org/jira/browse/HBASE-17581
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBase-17581-V1.patch
>
>
> mvn clean test -PskipXXXTests will skip all tests in XXX  module,  it works 
> for hbase-server, when we use mvn clean test -PskipServerTests, 
> it will skip all small and medium tests in hbase-server module. 
> However for some module like hbase-common, this command only skip small 
> tests, the medium tests are still executed.
> Need to ensure all components work in the same way



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17547) HBase-Spark Module : TableCatelog doesn't supports multiple columns from Single Column family

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840147#comment-15840147
 ] 

Jerry He commented on HBASE-17547:
--

+1

Changed the Affected version, since there is open source release with the 
hbase-spark module.

> HBase-Spark Module : TableCatelog doesn't supports multiple columns from 
> Single Column family
> -
>
> Key: HBASE-17547
> URL: https://issues.apache.org/jira/browse/HBASE-17547
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
> Attachments: HBASE-17547.master.001.patch
>
>
> Issue: HBase-Spark Module : TableCatelog doesn't supports multiple columns 
> from single column family.
> Description:
> Datasource API under HBase-Spark Module having error, which accessing more 
> than 1 columns from same column family.
> If your catalog having the format where you have multiple columns from single 
> / multiple column family, at that point it throws an exception, for example.
> def empcatalog = s"""{
> |"table":{"namespace":"empschema", "name":"emp"},
> |"rowkey":"key",
> |"columns":{
> |"empNumber":{"cf":"rowkey", "col":"key", "type":"string"},
> |"city":{"cf":"pdata", "col":"city", "type":"string"},
> |"empName":{"cf":"pdata", "col":"name", "type":"string"},
> |"jobDesignation":{"cf":"pdata", "col":"designation", "type":"string"},
> |"salary":{"cf":"pdata", "col":"salary", "type":"string"}
> |}
> |}""".stripMargin
> Here, we have city, name, designation, salary from pdata column family.
> Exception while saving Dataframe at HBase.
> java.lang.IllegalArgumentException: Family 'pdata' already exists so cannot 
> be added
> at 
> org.apache.hadoop.hbase.HTableDescriptor.addFamily(HTableDescriptor.java:827)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$createTable$1.apply(HBaseRelation.scala:98)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$createTable$1.apply(HBaseRelation.scala:95)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.createTable(HBaseRelation.scala:95)
> at 
> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:58)
> at 
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
> HBaseTableCatalog.scala class has getColumnFamilies method which returns 
> duplicates, which should not return.
> Unit test has been written for the same at DefaultSourceSuite.scala, 
> writeCatalog object definition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17547) HBase-Spark Module : TableCatelog doesn't supports multiple columns from Single Column family

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17547:
-
Affects Version/s: (was: 1.1.8)

> HBase-Spark Module : TableCatelog doesn't supports multiple columns from 
> Single Column family
> -
>
> Key: HBASE-17547
> URL: https://issues.apache.org/jira/browse/HBASE-17547
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
> Attachments: HBASE-17547.master.001.patch
>
>
> Issue: HBase-Spark Module : TableCatelog doesn't supports multiple columns 
> from single column family.
> Description:
> Datasource API under HBase-Spark Module having error, which accessing more 
> than 1 columns from same column family.
> If your catalog having the format where you have multiple columns from single 
> / multiple column family, at that point it throws an exception, for example.
> def empcatalog = s"""{
> |"table":{"namespace":"empschema", "name":"emp"},
> |"rowkey":"key",
> |"columns":{
> |"empNumber":{"cf":"rowkey", "col":"key", "type":"string"},
> |"city":{"cf":"pdata", "col":"city", "type":"string"},
> |"empName":{"cf":"pdata", "col":"name", "type":"string"},
> |"jobDesignation":{"cf":"pdata", "col":"designation", "type":"string"},
> |"salary":{"cf":"pdata", "col":"salary", "type":"string"}
> |}
> |}""".stripMargin
> Here, we have city, name, designation, salary from pdata column family.
> Exception while saving Dataframe at HBase.
> java.lang.IllegalArgumentException: Family 'pdata' already exists so cannot 
> be added
> at 
> org.apache.hadoop.hbase.HTableDescriptor.addFamily(HTableDescriptor.java:827)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$createTable$1.apply(HBaseRelation.scala:98)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation$$anonfun$createTable$1.apply(HBaseRelation.scala:95)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at 
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.createTable(HBaseRelation.scala:95)
> at 
> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:58)
> at 
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
> HBaseTableCatalog.scala class has getColumnFamilies method which returns 
> duplicates, which should not return.
> Unit test has been written for the same at DefaultSourceSuite.scala, 
> writeCatalog object definition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-17502:


Assignee: Jerry He

> Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support 
> matrix
> 
>
> Key: HBASE-17502
> URL: https://issues.apache.org/jira/browse/HBASE-17502
> Project: HBase
>  Issue Type: Task
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17502.patch, HBASE-17502-v2.patch
>
>
> Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
> HADOOP-10786 fixed the problem in Hadoop 2.6.1.
> Let's document it in the Hadoop support matrix.
> This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-17502.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0

Pushed.
Thanks for the review, Ted.

> Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support 
> matrix
> 
>
> Key: HBASE-17502
> URL: https://issues.apache.org/jira/browse/HBASE-17502
> Project: HBase
>  Issue Type: Task
>Reporter: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17502.patch, HBASE-17502-v2.patch
>
>
> Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
> HADOOP-10786 fixed the problem in Hadoop 2.6.1.
> Let's document it in the Hadoop support matrix.
> This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17502:
-
Attachment: HBASE-17502-v2.patch

Addressed Ted's comment.

> Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support 
> matrix
> 
>
> Key: HBASE-17502
> URL: https://issues.apache.org/jira/browse/HBASE-17502
> Project: HBase
>  Issue Type: Task
>Reporter: Jerry He
> Attachments: HBASE-17502.patch, HBASE-17502-v2.patch
>
>
> Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
> HADOOP-10786 fixed the problem in Hadoop 2.6.1.
> Let's document it in the Hadoop support matrix.
> This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17502:
-
Attachment: HBASE-17502.patch

> Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support 
> matrix
> 
>
> Key: HBASE-17502
> URL: https://issues.apache.org/jira/browse/HBASE-17502
> Project: HBase
>  Issue Type: Task
>Reporter: Jerry He
> Attachments: HBASE-17502.patch
>
>
> Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
> HADOOP-10786 fixed the problem in Hadoop 2.6.1.
> Let's document it in the Hadoop support matrix.
> This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

Jerry He created HBASE-17502:


 Summary: Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem 
in our hadoop support matrix
 Key: HBASE-17502
 URL: https://issues.apache.org/jira/browse/HBASE-17502
 Project: HBase
  Issue Type: Task
Reporter: Jerry He


Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
HADOOP-10786 fixed the problem in Hadoop 2.6.1.
Let's document it in the Hadoop support matrix.

This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-15597) Clean up configuration keys used in hbase-spark module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-15597:


Assignee: Jerry He

> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Critical
> Fix For: 2.0.0
>
>
> This should be considered a blocker for backport to branch-1 since it will 
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since 
> our configurations keys for the spark integration all relate to that system, 
> the prefix for all configuration keys (excluding those cases where we need to 
> do something special due to restrictions in how properties are handled by 
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we 
> should review all of our configuration keys to make sure they either conform 
> to the "hbase.spark" prefix or they have a comment documenting why they need 
> to be otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14375) define public API for spark integration module

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-14375:


Assignee: Jerry He

> define public API for spark integration module
> --
>
> Key: HBASE-14375
> URL: https://issues.apache.org/jira/browse/HBASE-14375
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Reporter: Sean Busbey
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
>
> before we can put the spark integration module into a release, we need to 
> annotate its public api surface.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17461) HBase shell "major_compact" command should properly convert "table_or_region_name" parameter to java byte array properly before simply calling "HBaseAdmin.majorCo

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821951#comment-15821951
 ] 

Jerry He edited comment on HBASE-17461 at 1/13/17 4:28 PM:
---

There is a similar JIRA HBASE-8865. 


was (Author: jinghe):
There is a similar JIRA HBASE-17461. 

> HBase shell "major_compact" command should properly convert 
> "table_or_region_name" parameter to java byte array properly before simply 
> calling "HBaseAdmin.majorCompact" method
> ---
>
> Key: HBASE-17461
> URL: https://issues.apache.org/jira/browse/HBASE-17461
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>
> On HBase shell, *major_compact* command simply passes the received 
> *table_or_region_name* parameter straight to java *HBaseAdmin.majorCompact* 
> method.
> On some corner cases, HBase tables row keys may have special characters. 
> Then, if a region is split in such a way that row keys with special 
> characters are now part of the region name, calling *major_compact* on this 
> regions will fail, if the special character ASCII code is higher than 127. 
> This happens because Java byte type is signed, while ruby byte type isn't, 
> causing the region name to be converted to a wrong string at Java side.
> For example, considering a region named as below:
> {noformat}
> test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1481745228583.b4bc69356d89018bfad3ee106b717285.
> {noformat} 
> Calling major_compat on it fails as follows:
> {noformat}
> hbase(main):008:0* major_compact 
> "test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1484177359169.8128fa75ae0cd4eba38da2667ac8ec98."
> ERROR: Illegal character code:44, <,> at 4. User-space table qualifiers can 
> only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: test,�B2!$�
> �G���1484177359169.8128fa75ae0cd4eba38da2667ac8ec98.
> {noformat}
> An easy solution is to convert *table_or_region_name* parameter properly, 
> prior to calling *HBaseAdmin.majorCompact* in the same way as it's already 
> done on some other shell commands, such as *get*:
> {noformat}
> admin.major_compact(table_or_region_name.to_s.to_java_bytes, family)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17461) HBase shell "major_compact" command should properly convert "table_or_region_name" parameter to java byte array properly before simply calling "HBaseAdmin.majorCompact

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821951#comment-15821951
 ] 

Jerry He commented on HBASE-17461:
--

There is a similar JIRA HBASE-17461. 

> HBase shell "major_compact" command should properly convert 
> "table_or_region_name" parameter to java byte array properly before simply 
> calling "HBaseAdmin.majorCompact" method
> ---
>
> Key: HBASE-17461
> URL: https://issues.apache.org/jira/browse/HBASE-17461
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>
> On HBase shell, *major_compact* command simply passes the received 
> *table_or_region_name* parameter straight to java *HBaseAdmin.majorCompact* 
> method.
> On some corner cases, HBase tables row keys may have special characters. 
> Then, if a region is split in such a way that row keys with special 
> characters are now part of the region name, calling *major_compact* on this 
> regions will fail, if the special character ASCII code is higher than 127. 
> This happens because Java byte type is signed, while ruby byte type isn't, 
> causing the region name to be converted to a wrong string at Java side.
> For example, considering a region named as below:
> {noformat}
> test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1481745228583.b4bc69356d89018bfad3ee106b717285.
> {noformat} 
> Calling major_compat on it fails as follows:
> {noformat}
> hbase(main):008:0* major_compact 
> "test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1484177359169.8128fa75ae0cd4eba38da2667ac8ec98."
> ERROR: Illegal character code:44, <,> at 4. User-space table qualifiers can 
> only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: test,�B2!$�
> �G���1484177359169.8128fa75ae0cd4eba38da2667ac8ec98.
> {noformat}
> An easy solution is to convert *table_or_region_name* parameter properly, 
> prior to calling *HBaseAdmin.majorCompact* in the same way as it's already 
> done on some other shell commands, such as *get*:
> {noformat}
> admin.major_compact(table_or_region_name.to_s.to_java_bytes, family)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814027#comment-15814027
 ] 

Jerry He commented on HBASE-17437:
--

The hbase.rootdir has a couple of other things, like the version file and 
clusterId file. Do these need to be duplicated in the new wal root dir? Any 
relevant checking that needs to be done? For example, this hbase.rootdir and 
this wal root dir is not a mismatch.

> Support specifying a WAL directory outside of the root directory
> 
>
> Key: HBASE-17437
> URL: https://issues.apache.org/jira/browse/HBASE-17437
> Project: HBase
>  Issue Type: Improvement
>  Components: Filesystem Integration, wal
>Affects Versions: 1.2.4
>Reporter: Yishan Yang
>Assignee: Yishan Yang
>  Labels: patch
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch
>
>
> Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some 
> FileSystems (such as Amazon S3) don’t support append or consistent writes. 
> These two properties are imperative for the WAL in order to avoid loss of 
> writes. However, StoreFiles don’t necessarily need the same consistency 
> guarantees (since writes are cached locally and if writes fail, they can 
> always be replayed from the WAL).
>  
> This JIRA aims to allow users to configure a log directory (for WALs) that is 
> outside of the root directory or even in a different FileSystem. The default 
> value will still put the log directory under the root directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17430) dead links in ref guide to class javadocs that moved out of user APIs.

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17430:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

Pushed. Thanks for the patch.

> dead links in ref guide to class javadocs that moved out of user APIs.
> --
>
> Key: HBASE-17430
> URL: https://issues.apache.org/jira/browse/HBASE-17430
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, website
>Reporter: Sean Busbey
>Assignee: Jan Hentschel
>  Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17430.master.001.patch
>
>
> The ref guide currently has one or more links to some javadocs that are no 
> longer in our user apis. unfortunately they link via a google search which 
> prevents the dead link detector from finding them.
> e.g. in architecture.adoc
> {code}
> Instead, they create small files similar to symbolic link files, named 
> link:http://www.google.com/url?q=http%3A%2F%2Fhbase.apache.org%2Fapidocs%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fio%2FReference.html=D=1=AFQjCNEkCbADZ3CgKHTtGYI8bJVwp663CA[Reference
>  files], which point to either the top or bottom part of the parent store 
> file according to the split point.
> {code}
> We should instead directly link to the correct location, e.g. 
> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/Reference.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810667#comment-15810667
 ] 

Jerry He commented on HBASE-17435:
--

+1 on v3.

> Call to preCommitStoreFile() hook encounters SaslException in secure 
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17435.branch-1.v2.txt, 17435.v1.txt, 17435.v2.txt, 
> 17435.v3.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where 
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at 
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)
> ... 16 more
> Caused by: javax.security.sasl.SaslException: GSS 

[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15809966#comment-15809966
 ] 

Jerry He commented on HBASE-16179:
--

Hi, [~tedyu]

I'll see if I can take a look here.

> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, 
> 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 
> 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806970#comment-15806970
 ] 

Jerry He commented on HBASE-17435:
--

Please enclose the new addition with a if 
(userProvider.isHBaseSecurityEnabled()).

> Call to preCommitStoreFile() hook encounters SaslException in secure 
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17435.v1.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where 
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at 
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)
> ... 16 more
> Caused by: 

[jira] [Comment Edited] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806970#comment-15806970
 ] 

Jerry He edited comment on HBASE-17435 at 1/7/17 6:55 AM:
--

Please enclose the new addition with if (userProvider.isHBaseSecurityEnabled()).


was (Author: jinghe):
Please enclose the new addition with a if 
(userProvider.isHBaseSecurityEnabled()).

> Call to preCommitStoreFile() hook encounters SaslException in secure 
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
>  Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17435.v1.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where 
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at 
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
> at 
> 

[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806206#comment-15806206
 ] 

Jerry He commented on HBASE-17435:
--

You can also get the server user's credentials and call ugi.addCredentials() 
the same way you add the token. Save a token.
I am not sure which way is more desirable.

> Call to preCommitStoreFile() hook encounters SaslException in secure 
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 17435.v1.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where 
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at 
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)
> ... 16 more
> Caused by: 

[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806097#comment-15806097
 ] 

Jerry He commented on HBASE-17435:
--

Yeah.  The ugi and User obtained from Rpc on the server side does not contain 
any Kerberos credentials.  They need to be added explicitly to be used by 
ugi.doAs() for that ugi.

> Call to preCommitStoreFile() hook encounters SaslException in secure 
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 17435.v1.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where 
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at 
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most 
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
> at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)
> ... 16 more
> Caused by: 

[jira] [Updated] (HBASE-17390) Online update of configuration for all servers leaves out masters

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17390:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to branch-1 and master. Thanks for the patch.

> Online update of configuration for all servers leaves out masters
> -
>
> Key: HBASE-17390
> URL: https://issues.apache.org/jira/browse/HBASE-17390
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Lars George
>Assignee: Jan Hentschel
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17390.master.001.patch, 
> HBASE-17390.master.002.patch, HBASE-17390.master.003.patch, 
> HBASE-17390.master.004.patch
>
>
> Looking at the admin API and this method
> {code}
>   public void updateConfiguration() throws IOException {
> for (ServerName server : this.getClusterStatus().getServers()) {
>   updateConfiguration(server);
> }
>   }
> {code}
> you can see that it calls {{getServers()}} which only returns the region 
> servers.
> What is missing is also calling on {{getMaster()}} and {{getBackupMasters()}} 
> to also send them a signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17390) Online update of configuration for all servers leaves out masters

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799332#comment-15799332
 ] 

Jerry He commented on HBASE-17390:
--

+1

> Online update of configuration for all servers leaves out masters
> -
>
> Key: HBASE-17390
> URL: https://issues.apache.org/jira/browse/HBASE-17390
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Lars George
>Assignee: Jan Hentschel
> Attachments: HBASE-17390.master.001.patch, 
> HBASE-17390.master.002.patch, HBASE-17390.master.003.patch, 
> HBASE-17390.master.004.patch
>
>
> Looking at the admin API and this method
> {code}
>   public void updateConfiguration() throws IOException {
> for (ServerName server : this.getClusterStatus().getServers()) {
>   updateConfiguration(server);
> }
>   }
> {code}
> you can see that it calls {{getServers()}} which only returns the region 
> servers.
> What is missing is also calling on {{getMaster()}} and {{getBackupMasters()}} 
> to also send them a signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796881#comment-15796881
 ] 

Jerry He commented on HBASE-16010:
--

This is good plan, if I missed it earlier!

Ok. let us change the names then.

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reassigned HBASE-16010:


Assignee: Jerry He  (was: Matt Warhaftig)

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17370) Fix or provide shell scripts to drain and decommission region server

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796812#comment-15796812
 ] 

Jerry He commented on HBASE-17370:
--

There are existing ruby scrips (draining_servers.rb, graceful_stop.sh).  We can 
discuss what we can do about them (update or fold them into shell commands).


> Fix or provide shell scripts to drain and decommission region server
> 
>
> Key: HBASE-17370
> URL: https://issues.apache.org/jira/browse/HBASE-17370
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>
> 1. Update the existing shell scripts to use the new drain related API.
> 2  Or provide new shell scripts.
> 3. Provide a 'decommission' shell tool that puts the server in drain mode and 
> offload the server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796779#comment-15796779
 ] 

Jerry He commented on HBASE-16010:
--

bq. Would be great to use Enis's above suggestion. Instead of 
drainRegionServers*/removeDrainFromRegionServers* in functions and PBs, 
decommissionRegionServers*/recommissionRegionServers* would have been better.
There is currently a slight difference between 'drain mode' and 'decommission' 
from what I understand.  'decommission' uses the region mover to offload the 
regions in addition to putting the server on 'drain mode', where the drain API 
only does the latter.
I think we should just provide the drain API as they are in the patch.  The 
java doc of the API clearly stated their purpose.
Then we can have a single shell command or script that provides the convenient 
'decommission' feature.   A subtask was created to do that.
How does it sound? [~enis] and [~appy]


> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17390) Online update of configuration for all servers leaves out masters

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15791649#comment-15791649
 ] 

Jerry He commented on HBASE-17390:
--

lgtm. Could you add a test case?

> Online update of configuration for all servers leaves out masters
> -
>
> Key: HBASE-17390
> URL: https://issues.apache.org/jira/browse/HBASE-17390
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Lars George
>Assignee: Jan Hentschel
> Attachments: HBASE-17390.master.001.patch
>
>
> Looking at the admin API and this method
> {code}
>   public void updateConfiguration() throws IOException {
> for (ServerName server : this.getClusterStatus().getServers()) {
>   updateConfiguration(server);
> }
>   }
> {code}
> you can see that it calls {{getServers()}} which only returns the region 
> servers.
> What is missing is also calling on {{getMaster()}} and {{getBackupMasters()}} 
> to also send them a signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)