[
https://issues.apache.org/jira/browse/DRILL-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009376#comment-17009376
]
ASF GitHub Bot commented on DRILL-7503:
---------------------------------------
paul-rogers commented on issue #1944: DRILL-7503: Refactor the project operator
URL: https://github.com/apache/drill/pull/1944#issuecomment-571437864
@ihuzenko, refactored some additional steps to adopt the solution you
suggested. This version still uses the interface, with an implementation in a
class other than the project record batch. This turns out to be handy because,
oddly, the project operator generates code for two separate incoming batches:
the "real" one and a "fake" empty one. The `ProjectRecordBatchBuilder` holds
onto the input batch so we don't have to pass it into the materializer and back
out.
This version tries to eliminate all references to the incoming batch in the
materializer, and instead work with the batch schema. Annoyingly, the
`ExpressionTreeMaterializer` needs the input batch so it can iterate over the
vectors to get their schemas. If all we need is the schema, we don't need
actual vectors. So, if we can pass in a schema, we can completely separate code
gen from physical vectors.
The next refactoring move is to change this code to work with a schema (or
interface to obtain the schema) rather than the actual vectors. Now, as it
turns out, the batch schema has limitations for complex types, which is one of
the reasons we created the `TupleMetadata` family of classes. So, perhaps we
can convert the incoming batch to a `TupleMetadata` schema and use that. (The
code to do that already exists in the `RowSet` classes.)
Or, we can just pass an interface which will return the `TypedFieldId` for
each column. Or, do that conversion ahead of time, and pass in the results.
Will have to play with it some to see which solution is the simplest.
Since the required work will be rather large; I propose we do that as a
separate PR.
Have we done enough for this one PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Refactor project operator
> -------------------------
>
> Key: DRILL-7503
> URL: https://issues.apache.org/jira/browse/DRILL-7503
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.17.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Minor
> Fix For: 1.18.0
>
>
> Work on another ticket revealed that the Project operator ("record batch")
> has grown quite complex. The setup phase lives in the operator as one huge
> function. The function combines the "logical" tasks of working out the
> projection expressions and types, the code gen for those expressions, and the
> physical setup of vectors.
> The refactoring breaks up the logic so that it is easier to focus on the
> specific bits of interest.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
