[
https://issues.apache.org/jira/browse/HIVE-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660950#action_12660950
]
Joydeep Sen Sarma commented on HIVE-207:
----------------------------------------
one thing about the SerDe/Objectinspector stuff is that the schema is dictated
by the serde. Essentially - the DDL in hive is just a way to create
configuration for the native serde (dynamic serde). At this point - what the
serde returns whatever the DDL defines.
However - the DDL is not required for all tables. Tables just need to have a
SerDe - and the schema will be obtained from the Serde. There is a create table
command that just takes in a serde specification (although i can't recount ths
syntax off the bat - and there may be issues). We have lots of custom formatted
tables in our environment for which the schema is not obtained from a DDL - but
from a serde implementation.
The objectinspector stuff is also complicated because we support complex and
nested types. So the ObjectInspector interfaces are somewhat similar to Java
Reflection apis and are recursive.
Regarding the specific proposal for the columnset - i think this is
implementable inside the objectinspector framework. I take it that the data
model is a flat set of columns. the deserialize() implementation will just
populate the equivalent of the columnset structure (which is part of ur
implementation) and will return a container with reference to the underlying
serialized buffer and the columnset structure. You would have to implement a
StructObjectInspector (which is what the getObjectInspector() should return).
If u look at the methods in this (comments on what the implementation might
have to do):
public List<? extends StructField> getAllStructFieldRefs();
// this is just getTableColumnNames() from ur columnset struct
public StructField getStructFieldRef(String fieldName);
// this is whatever is required to extract a field from a underlying buffer
- for example some offset or index
public Object getStructFieldData(Object data, StructField fieldRef);
// this actually retrieves the field from the buffer. At this point - you can
used information about used/unused columns to return nulls as required.
public List<Object> getStructFieldsDataAsList(Object data);
// this is just a transformation function - i am not entirely sure when this
is invoked - but the implementation is obvious
hope this explains things somewhat .. (unfortunately the design/scope of the
serde stuff is not that well documented ..)
> Change SerDe API to allow skipping unused columns
> -------------------------------------------------
>
> Key: HIVE-207
> URL: https://issues.apache.org/jira/browse/HIVE-207
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor, Serializers/Deserializers
> Reporter: David Phillips
>
> A deserializer shouldn't have to deserialize columns that are never used by
> the query processor. A serializer shouldn't have to examine unused columns
> that are known to always be null.
> As an example, we store data as a Protocol Buffer structure with ~60 fields.
> Running a "select count(1)" currently requires deserializing all fields,
> which includes checking if they exist and formatting the data appropriately.
> This is expensive and unnecessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
