[
https://issues.apache.org/jira/browse/HIVE-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661859#action_12661859
]
Joydeep Sen Sarma commented on HIVE-207:
----------------------------------------
@Zheng - i finally grokked at the Dynamic SerDe code a bit. as i suspected - it
reads all the fields in one shot in the deserialize code. this is natural given
that the protocol is a thrift protocol and thrift reads all fields in one shot.
while it can skip fields - the skipping itself has a cost (ie. if there was a
format smart enough to store offsets for each field and go directly to a
particular field - dynamic serde would not be able to leverage it).
i did mean what Prasad inferred. that whatever data structures are used to
represent type information from the ddl are exposed via a public interface. an
additional method call in the serde can be optionally implemented that can take
this data structure in and popularte configuration keys (that can then be
stored as serdeproperties). it would be trivial to convert the dynamic serde
<-> DDL interaction to follow this model as well.
(will try to educate myself a bit on protocol buffers - a jist would be greatly
appreciated though - especially as far as how the serialization lends itself to
efficient retrieval of specific fields?)
> Change SerDe API to allow skipping unused columns
> -------------------------------------------------
>
> Key: HIVE-207
> URL: https://issues.apache.org/jira/browse/HIVE-207
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor, Serializers/Deserializers
> Reporter: David Phillips
>
> A deserializer shouldn't have to deserialize columns that are never used by
> the query processor. A serializer shouldn't have to examine unused columns
> that are known to always be null.
> As an example, we store data as a Protocol Buffer structure with ~60 fields.
> Running a "select count(1)" currently requires deserializing all fields,
> which includes checking if they exist and formatting the data appropriately.
> This is expensive and unnecessary.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
