[
https://issues.apache.org/jira/browse/HBASE-14801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189831#comment-15189831
]
Zhan Zhang commented on HBASE-14801:
------------------------------------
[~jerryhe] The schema is defined on the application level. The application can
have the catalog persisted in some storage or defined directly in the code.
Following is a short description.
1. table defined the hbase table to operate on.
2. in the columns: the first dimension defined the column name in spark-sql,
second dimension(cf) defined the column family correspondingly in hbase table,
and third dimension (col) defined the column qualifier, and type defines the
datatype in sparksql, serdes defined the customized serdes.
3. rowkey defined the composite key in the hbase key part, separated by ":",
correspondingly the "cf" part has to be defined as "rowkey".
val catalog = s"""{
|"table":{"namespace":"default", "name":"htable"},
|"rowkey":"key1:key2",
|"columns":{
|"col1":{"cf":"rowkey", "col":"key1", "type":"string"},
|"col2":{"cf":"rowkey", "col":"key2", "type":"double"},
|"col3":{"cf":"cf1", "col":"col2", "type":"binary"},
|"col4":{"cf":"cf1", "col":"col3", "type":"timestamp"},
|"col5":{"cf":"cf1", "col":"col4", "type":"double",
"serdes":"${classOf[DoubleSerDes].getName}"},
|"col6":{"cf":"cf1", "col":"col5", "type":"$map"},
|"col7":{"cf":"cf1", "col":"col6", "type":"$array"},
|"col8":{"cf":"cf1", "col":"col7", "type":"$arrayMap"}
|}
|}""".stripMargin
> Enhance the Spark-HBase connector catalog with json format
> ----------------------------------------------------------
>
> Key: HBASE-14801
> URL: https://issues.apache.org/jira/browse/HBASE-14801
> Project: HBase
> Issue Type: Sub-task
> Reporter: Zhan Zhang
> Assignee: Zhan Zhang
> Attachments: HBASE-14801-1.patch, HBASE-14801-2.patch,
> HBASE-14801-3.patch, HBASE-14801-4.patch, HBASE-14801-5.patch,
> HBASE-14801-6.patch, HBASE-14801-7.patch, HBASE-14801-8.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
