[
https://issues.apache.org/jira/browse/HIVE-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707160#comment-15707160
]
Sergey Shelukhin commented on HIVE-14453:
-----------------------------------------
Different kinds of streams are not distinguished for data streams, only index
streams.
Protobufs are necessary for two reasons:
1) Only the physical writer knows certain things that are put in them, e.g.
file offsets.
2) Alternative writers do not have to serialize protobufs as bytes (or at all).
An example implementation of using ORC as cache storage format for text tables
is in the blocked JIRA that now has a WIP patch.
I think it's actually no more brittle than any other separation of concerns -
it just separates the physical file on disk and its organization from the logic
of writing the data and metadata, which is good even if it reduces flexibility
somewhat, as it avoids monolithic dependencies for these two unrelated things.
> refactor physical writing of ORC data and metadata to FS from the logical
> writers
> ---------------------------------------------------------------------------------
>
> Key: HIVE-14453
> URL: https://issues.apache.org/jira/browse/HIVE-14453
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HIVE-14453.01.patch, HIVE-14453.02.patch,
> HIVE-14453.03.patch, HIVE-14453.patch
>
>
> ORC data doesn't have to go directly into an HDFS stream via buffers, it can
> go somewhere else (e.g. a write-thru cache, or an addressable system that
> doesn't require the stream blocks to be held in memory before writing them
> all together).
> To that effect, it would be nice to abstract the data block/metadata
> structure creating from the physical file concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)