[
https://issues.apache.org/jira/browse/TRAFODION-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715434#comment-14715434
]
Suresh Subbiah commented on TRAFODION-1462:
-------------------------------------------
The serialization change was done by Anoop prior to Trafodion 1.0. We were
running developer regresssions with the feature enabled but the feature was
enabled by default.
I will summarize what I understand about the feature here. Please ask questions
and we can collectively fill in parts I have not explained.
Serialization has as its goal, the ability to evaluate simple predicates
(equality, range, and possibly a few others) in the HBase Region Server using
HBase Filters. In addition to creating the appropriate HBase filters (a list of
SingleColumnValueFilters with the relevant compareOps) and associating it with
the scanner, we also need to make sure that data in HBase is in a format that
will allow simple byte comparison. Several Trafodion datatypes are not stored
in disk in a manner that will allow byte comparison for range predicates. Now
this is true only when the column is not part of the key. In the key, a column
is always stored in a format such that the binary representation sorts in an
identical manner to the actual value of that column (i.e. regardless of
serialization). A lot of work in the Trafodion stack for the serialization
feature is to ensure that data is appropriately formatted before it is sent to
HBase and deserialized as it is read from HBase.
Not all SQL types can currently be serialized. Of the types that can be
serialized some like a fixed length ISO88591 CHAR column need no further
encoding/formatting, since the data already sorts in binary representation in
the same manner as it sorts in value. Other types like a signed integer need
encoding (or serialization) so that binary data sorts as required.
character type : can be serialized. Varchar needs encoding, UCS2 needs
encoding, other char types do not need encoding
numeric type: short, int, largeint can be serialized. Float, bignum, decimal
not serialized. Little endian machines do not need encoding. Unsigned types
also do not need encoding.
datetime, interval types : not serialized
lob (not serialized when treat_blob_as_varchar is OFF, otherwise it is) .
Serialized data format is also not compatible with aligned row format. With
aligned rows, the start offset for a SQL column can vary from row to row and
all SQL columns correspond to a single column in HBase. Therefore simple HBase
Filters cannot be used to evaluate predicates on a given column.
> Enable HBase serialization by default
> -------------------------------------
>
> Key: TRAFODION-1462
> URL: https://issues.apache.org/jira/browse/TRAFODION-1462
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
> Affects Versions: 1.0 (pre-incubation)
> Reporter: Suresh Subbiah
> Assignee: Suresh Subbiah
> Fix For: 2.0-incubating
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently when a Trafodion table is created, by default data stored in HBase
> tables is not serialized (i.e. not encoded or stored in a form that will
> allow evaluation of range predicates with HBase Filters executing in the
> RegionServer proceses). Trafodion did support storing Trafodion data in such
> an encoded format, but the feature had to be explicitly enabled with
> cqd HBASE_SERIALIZATION 'ON' ;
> With this change the default setting for the cqd has been changed to ON.
> Tables will now be created by default with columns having the serialized
> attribute. This can be verified by using the showddl command. For example
> >>showddl customer_demographics ;
> CREATE TABLE TRAFODION.SCH.CUSTOMER_DEMOGRAPHICS
> (
> CD_DEMO_SK INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , CD_GENDER CHAR(1) CHARACTER SET ISO88591 COLLATE
> DEFAULT DEFAULT NULL SERIALIZED
> , CD_MARITAL_STATUS CHAR(1) CHARACTER SET ISO88591 COLLATE
> DEFAULT DEFAULT NULL SERIALIZED
> , CD_EDUCATION_STATUS CHAR(20) CHARACTER SET ISO88591 COLLATE
> DEFAULT DEFAULT NULL SERIALIZED
> , CD_PURCHASE_ESTIMATE INT DEFAULT NULL SERIALIZED
> , CD_CREDIT_RATING CHAR(10) CHARACTER SET ISO88591 COLLATE
> DEFAULT DEFAULT NULL SERIALIZED
> , CD_DEP_COUNT INT DEFAULT NULL SERIALIZED
> , CD_DEP_EMPLOYED_COUNT INT DEFAULT NULL SERIALIZED
> , CD_DEP_COLLEGE_COUNT INT DEFAULT NULL SERIALIZED
> )
> STORE BY (CD_DEMO_SK ASC)
> ;
> CREATE INDEX CD_DEP_COLLEGE_COUNT_IDX ON TRAFODION.SCH.CUSTOMER_DEMOGRAPHICS
> (
> CD_DEP_COLLEGE_COUNT ASC
> )
> ;
> CREATE INDEX CD_DEP_COUNT_IDX ON TRAFODION.SCH.CUSTOMER_DEMOGRAPHICS
> (
> CD_DEP_COUNT ASC
> )
> ;
> The create table statements used for this table is
> create table customer_demographics
> (
> cd_demo_sk int not null,
> cd_gender char(1),
> cd_marital_status char(1),
> cd_education_status char(20),
> cd_purchase_estimate int,
> cd_credit_rating char(10),
> cd_dep_count int,
> cd_dep_employed_count int,
> cd_dep_college_count int
> ) store by (cd_demo_sk);
> create index cd_dep_count_IDX on customer_demographics(cd_dep_count);
> create index cd_dep_college_count_IDX on
> customer_demographics(cd_dep_college_count);
> The indexes are also serialized though showddl does not show it.
> Having a column as serialized allows data to be stored in a format suitable
> for using HBase filters. However to get HBase filters in the execution plan
> for a SQL query, the cqd HBASE_FILTER_PREDS must be set to 'ON'. This cqd
> still defaults to 'OFF' as in some cases it is better to evaluate the
> predicate in the Trafodion/SQL layer, rather than the RegionServers (since
> Trafodion can use more ESPs to further parallelize predicate evaulation).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
