[ 
https://issues.apache.org/jira/browse/DRILL-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394714#comment-16394714
 ] 

ASF GitHub Bot commented on DRILL-6230:
---------------------------------------

GitHub user paul-rogers opened a pull request:

    https://github.com/apache/drill/pull/1161

    DRILL-6230: Extend row set readers to handle hyper vectors

    The current row set readers have incomplete support for hyper-vectors. To 
add full support, we need an interface that supports either single batches or 
hyper batches. Accessing vectors in hyper batches differs depending on whether 
the vector is at the top level or is nested. See this post for details. Also 
includes a simpler reader template: replaces the original three classes with 
one, in parallel with the writers.
    
    Key changes:
    
    * Refactor the readers to generate just the required reader, then build up 
the optional and repeated readers as layers on top of the generated reader. 
This is the same structure that the writers already use.
    * Add and test support for hyper-vectors.
    * Extend the existing "vector accessor" abstraction to fully support the 
highly complex process of locating nested vectors (those within a map or union) 
in a hyper-batch.
    * Introduce the idea of a "null state" abstraction to handle the messy null 
handling in unions and repeated lists.
    * Modifies tests as needed for the new internal format of vector readers.
    
    To keep the PR from getting overly large, this PR strips out the actual 
union and list support. That support will be added in a future PR. Similarly, 
there are matching changes to writers that will also be done in a separate PR.
    
    Other minor changes:
    
    * Revises the previous utility PR. In some cases, it turns out to be 
cleaner to use a separate `mapValue()` function instead of `objArray()`, even 
though both produce an object array. Calling it `mapValue()` makes it a bit 
clearer what we're trying to accomplish.
    
    This PR is not needed for Drill 1.13; it can go into Drill 1.14.
    
    See [this 
post](https://github.com/paul-rogers/drill/wiki/Batch-Handling-Upgrades) for 
details of the end-state toward which this PR is one step.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/paul-rogers/drill DRILL-6230

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1161
    
----
commit e9891c561088d2c79ab1758dc857a8a52ec253ac
Author: Paul Rogers <progers@...>
Date:   2018-03-11T07:43:36Z

    Accessor revisions

commit 6f6e3eb803793d71a5e8dba8362737bac66d923c
Author: Paul Rogers <progers@...>
Date:   2018-03-11T22:41:42Z

    Merge of exec row set readers & tests

commit 65cd6205ea8e85ac4e001634ffa24268a57ce273
Author: Paul Rogers <progers@...>
Date:   2018-03-12T00:23:35Z

    Fixed tests to remove work not in this PR

----


> Extend row set readers to handle hyper vectors
> ----------------------------------------------
>
>                 Key: DRILL-6230
>                 URL: https://issues.apache.org/jira/browse/DRILL-6230
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Major
>             Fix For: 1.14.0
>
>
> The current row set readers have incomplete support for hyper-vectors. To add 
> full support, we need an interface that supports either single batches or 
> hyper batches. Accessing vectors in hyper batches differs depending on 
> whether the vector is at the top level or is nested. SeeĀ [this 
> post|https://github.com/paul-rogers/drill/wiki/BH-Column-Readers] for 
> details. Also includes a simpler reader template: replaces the original three 
> classes with one, in parallel with the writers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to