[
https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sujith updated SPARK-26432:
---
Description:
Getting NoSuchMethodException :
org.apache.hadoop.hbase.security.token.TokenUtil(org.apache.hadoop.conf.Configuration)
while trying connect hbase 2.1 service from spark.
This is mainly happening because in spark uses a deprecated hbase api
public static Token obtainToken(Configuration
conf)
for obtaining the token and the same has been removed from hbase 2.1 version.
Test steps:
Steps to test Spark-Hbase connection
1. Create 2 tables in hbase shell
>Launch hbase shell
>Enter commands to create tables and load data
create 'table1','cf'
put 'table1','row1','cf:cid','20'
create 'table2','cf'
put 'table2','row1','cf:cid','30'
>Show values command
get 'table1','row1','cf:cid' will diplay value as 20
get 'table2','row1','cf:cid' will diplay value as 30
2.Run SparkHbasetoHbase class in testSpark.jar using spark-submit
spark-submit --master yarn-cluster --class
com.mrs.example.spark.SparkHbasetoHbase --conf
"spark.yarn.security.credentials.hbase.enabled"="true" --conf
"spark.security.credentials.hbase.enabled"="true" --keytab
/opt/client/user.keytab --principal sen testSpark.jar
The SparkHbasetoHbase class will update the value of table2 with sum of values
of table1 & table2.
table2 = table1+table2
3.Verify the result in hbase shell
Expected Result: The value of table2 should be 50.
get 'table1','row1','cf:cid' will diplay value as 50
Actual Result : Not updating the value as an error will be thrown when spark
tries to connect with hbase service.
Attached the snapshot of error logs below for more details
was:
Getting NoSuchMethodException :
org.apache.hadoop.hbase.security.token.TokenUtil(org.apache.hadoop.conf.Configuration)
while trying connect hbase 2.1 service from spark.
This is mainly happening because in spark uses a deprecated hbase api
public static Token obtainToken(Configuration
conf)
for obtaining the token and the same has been removed from hbase 2.1 version.
Attached the snapshot of error logs
> Not able to connect Hbase 2.1 service Getting NoSuchMethodException while
> trying to obtain token from Hbase 2.1 service.
>
>
> Key: SPARK-26432
> URL: https://issues.apache.org/jira/browse/SPARK-26432
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.2, 2.4.0
>Reporter: Sujith
>Priority: Major
> Attachments: hbase-dep-obtaintok.png
>
>
> Getting NoSuchMethodException :
> org.apache.hadoop.hbase.security.token.TokenUtil(org.apache.hadoop.conf.Configuration)
> while trying connect hbase 2.1 service from spark.
> This is mainly happening because in spark uses a deprecated hbase api
> public static Token obtainToken(Configuration
> conf)
> for obtaining the token and the same has been removed from hbase 2.1 version.
> Test steps:
> Steps to test Spark-Hbase connection
> 1. Create 2 tables in hbase shell
> >Launch hbase shell
> >Enter commands to create tables and load data
> create 'table1','cf'
> put 'table1','row1','cf:cid','20'
> create 'table2','cf'
> put 'table2','row1','cf:cid','30'
>
> >Show values command
> get 'table1','row1','cf:cid' will diplay value as 20
> get 'table2','row1','cf:cid' will diplay value as 30
>
>
> 2.Run SparkHbasetoHbase class in testSpark.jar using spark-submit
> spark-submit --master yarn-cluster --class
> com.mrs.example.spark.SparkHbasetoHbase --conf
> "spark.yarn.security.credentials.hbase.enabled"="true" --conf
> "spark.security.credentials.hbase.enabled"="true" --keytab
> /opt/client/user.keytab --principal sen testSpark.jar
> The SparkHbasetoHbase class will update the value of table2 with sum of
> values of table1 & table2.
> table2 = table1+table2
>
> 3.Verify the result in hbase shell
> Expected Result: The value of table2 should be 50.
> get 'table1','row1','cf:cid' will diplay value as 50
> Actual Result : Not updating the value as an error will be thrown when spark
> tries to connect with hbase service.
> Attached the snapshot of error logs below for more details
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
