[ 
https://issues.apache.org/jira/browse/HIVE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888279#comment-13888279
 ] 

Eric Hanson commented on HIVE-6234:
-----------------------------------

This is just getting started. I need to put this aside for a while (probably at 
least until the end of Feb.). I parked the latest information here on the JIRA.

> Implement fast vectorized InputFormat extension for text files
> --------------------------------------------------------------
>
>                 Key: HIVE-6234
>                 URL: https://issues.apache.org/jira/browse/HIVE-6234
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Eric Hanson
>            Assignee: Eric Hanson
>         Attachments: HIVE-6234.02.patch, HIVE-6234.03.patch, Vectorized Text 
> InputFormat design.docx, Vectorized Text InputFormat design.pdf, 
> state-diagram.jpg
>
>
> Implement support for vectorized scan input of text files (plain text with 
> configurable record and field separators). This should work for CSV files, 
> tab delimited files, etc. 
> The goal is to provide high-performance reading of these files using 
> vectorized scans, and also to do it as an extension of existing Hive. Then, 
> if vectorized query is enabled, existing tables based on text files will be 
> able to benefit immediately without the need to use a different input format. 
> After upgrading to new Hive bits that support this, faster, vectorized 
> processing over existing text tables should just work, when vectorization is 
> enabled.
> Another goal is to go beyond a simple layering of vectorized row batch 
> iterator over the top of the existing row iterator. It should be possible to, 
> say, read a chunk of data into a byte buffer (several thousand or even 
> million rows), and then read data from it into vectorized row batches 
> directly. Object creations should be minimized to save allocation time and GC 
> overhead. If it is possible to save CPU for values like dates and numbers by 
> caching the translation from string to the final data type, that should 
> ideally be implemented.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Reply via email to